create the new development scripts/library

[ohanar]

In order to switch to the new workflow, we need to create a set of development scripts for interfacing against git and trac. We use ​https://github.com/sagemath/sage-workflow/issues to keep track of issues.

[tkluck]

The dependencies got lost through the ./sage --dev upload script. I'm putting them back.

[tkluck]

Remove circular dependency.

[saraedum]

We can now add comments.

[saraedum]

I will spend the better part of this week trying to complete these scripts (i.e. mostly writing doctests). If anybody else wants to do substantial work on these scripts, it's probably best to contact me, so we don't work on the same issues.

[saraedum]

I did some work on doctesting and better reporting of exceptions. I will not be able to work on this for about a week but will try to finish this afterwards.

[vbraun]

There is a problem in the handling of untracked files. Git interprets "untracked" as untracked in cwd. In fact, just having been given --git-dir we can't possibly know where the working directory is. For example, consider the filesystem layout:

$ tree -a
.
├── a
│   ├── file_in_a
│   └── .git
│       ├── branches
│       [...] 
└── b
    └── file_in_b
$ cd b
$ git --git-dir=../a/.git ls-files --other
file_in_b

whereas you probably would have expected file_in_a.

[saraedum]

Replying to vbraun: This should be fixed now in git_interface.untracked_files().

[saraedum]

wget (when invoked by the dev scripts) complains that the Common Name for the SSL certificate on trac.sagemath.org refers to sagecell.sagemath.org. I don't know much about SSL but do we not have a certificate for trac.sagemath.org?

[vbraun]

The scripts should use curl instead of wget, the latter is not available on OSX since Apple is prejudiced against the GPL.

The SSL problem is that we are trying to host two SSL virtual domains on the same IP, that is a no-no. I've posted to sage-devel about it at ​https://groups.google.com/d/msg/sage-devel/VRohw4DSNrk/TpjWGQ-Vtu8J

[ohanar]

If you are trying to download a file from within python you should probably use the retrieve method of urllib.FancyURLopener. See src/bin/sage-download-file for an example.

[vbraun]

Also, I'm not entirely happy with the git interface. Though I also think its good enough to use in the git scripts and its probably a bad idea to do any architectural changes at this point. For the record I've built on the current code at ​https://github.com/vbraun/SageUI/tree/release/src/sageui/model. Changes incude

• Use proxy objects instead of having to import magic constants:

  In [3]: git.status()   # return stdout by default, the most desirable behaviour
  Out[3]: '# On branch dev_scripts\nnothing to commit, working directory clean\n'
  
  In [4]: git.exit_code.status()
  Out[4]: 0
  
  In [5]: git.silent.status()
  
  In [6]: git.echo.status()
  # On branch dev_scripts
  nothing to commit, working directory clean
  

• Don't hardcode the Sage repo directory, this is probably going to be useful elsewhere
• Use the GIT_DIR and GIT_WORK_TREE environment variables instead of --git-dir (though warning: there seems to be a bug in git status, it is not respecting GIT_WORK_TREE)
• Split the git interface (handling the mechanics of interfacing with the git binary) from the logical repository handling (like the create_ticket method)
• Put all the doctesting setup into a utility class instead of having lengthy sections creating Doctest* classes and sample repos.

[saraedum]

Replying to ohanar:

If you are trying to download a file from within python you should probably use the retrieve method of urllib.FancyURLopener. See src/bin/sage-download-file for an example.

Ok. I changed it to use urllib.

[saraedum]

Replying to vbraun:

• Use proxy objects instead of having to import magic constants

Right, this is syntactically much nicer. I changed this in my branch.

• Don't hardcode the Sage repo directory, this is probably going to be useful elsewhere

Where is the repo hardcoded?

• Use the GIT_DIR and GIT_WORK_TREE environment variables instead of --git-dir (though warning: there seems to be a bug in git status, it is not respecting GIT_WORK_TREE)

Could you explain the benefit of GIT_DIR and GIT_WORK_TREE?

• Split the git interface (handling the mechanics of interfacing with the git binary) from the logical repository handling (like the create_ticket method)

create_ticket is not part of the git interface anymore. Are there any other methods you want to move?

• Put all the doctesting setup into a utility class instead of having lengthy sections creating Doctest* classes and sample repos.

Good point. I was not sure initially what the doctests would look like. Now it seems that there are only two cases (one user, two users). I'll put the usual setup into special functions.

[vbraun]

Thanks, looks much better.

I would say anything that accesses the UI or Config ought to not be part of the low-level git interface. That should only know about the path to the repo and nothing else, so it can be used in the future for other git repos. But its not really urgent to fine-tune this right now.

Functionally, setting GIT_DIR and GIT_WORK_TREE is basically equivalent to setting --git-dir and always chdir-ing to the right directory. I just find the former better structured, you don't have to filter out arguments for the log etc. But again its not a big difference.

[saraedum]

Replying to vbraun:

I would say anything that accesses the UI or Config ought to not be part of the low-level git interface. That should only know about the path to the repo and nothing else, so it can be used in the future for other git repos. But its not really urgent to fine-tune this right now.

Functionally, setting GIT_DIR and GIT_WORK_TREE is basically equivalent to setting --git-dir and always chdir-ing to the right directory. I just find the former better structured, you don't have to filter out arguments for the log etc. But again its not a big difference.

I agree with both points. It would be nice to change this some day. For the time being I would rather leave it the way it is, just to be able to focus on the missing bits in sagedev.py.

[ohanar]

Replying to vbraun:

Functionally, setting GIT_DIR and GIT_WORK_TREE is basically equivalent to setting --git-dir and always chdir-ing to the right directory.

Not quite, setting GIT_WORK_TREE doesn't behave the same as chdir-ing to a given directory, it simply specifies what git should consider is the root of work tree -- it will still display relative paths to the current directory (for all commands, as far as I can tell) if the current directory is a within the specified work tree.

[vbraun]

I have a DSA ssh key (~/.ssh/id_dsa and ~/.ssh/id_dsa.pub). I encountered two bugs:

1. The dev scripts only check for a RSA key, which I don't have.
2. Since it didn't find my key, the scripts generated a new one. But as (~/.ssh/id_rsa.pub and ~/.ssh/id_rsa.pub.pub), making the private key look like a public key that can be shared.

[saraedum]

Thanks for trying things out. This should be fixed now.

Replying to vbraun:

I have a DSA ssh key (~/.ssh/id_dsa and ~/.ssh/id_dsa.pub). I encountered two bugs:

1. The dev scripts only check for a RSA key, which I don't have.
2. Since it didn't find my key, the scripts generated a new one. But as (~/.ssh/id_rsa.pub and ~/.ssh/id_rsa.pub.pub), making the private key look like a public key that can be shared.

[vbraun]

Thanks!

There is a minor annoyance left, I have already set up ssh keys yet the dev scripts insist on doing it again and prompt for username/password. We should first check if authetication to gitolite works, and only upload keys if it doesn't. I suggest the following test: Run

ssh -o "StrictHostKeyChecking yes" -o "PasswordAuthentication no" -o "PubkeyAuthentication yes" -p 2222 git@trac.sagemath.org help

If the exit code is 0, then the host key and ssh keys are already set up.

[saraedum]

Replying to vbraun:

There is a minor annoyance left, I have already set up ssh keys yet the dev scripts insist on doing it again and prompt for username/password. We should first check if authetication to gitolite works, and only upload keys if it doesn't. I suggest the following test: Run

I now simply changed the scripts to only ask the user once. Your solution is much nicer but I just did something that is easy to implement. (Also, I'm not sure how portable the ssh call is, and I do not have a MAC to test.)

[saraedum]

The SavingDicts?' data is stored in DOT_SAGE. This is not a good idea because multiple copies of sage will write to the same data file. I guess these dicts need to go somewhere in the repository. Where should that be? local/?

[tkluck]

maybe .git is best. They contain data about branches, right? I would also expect to be able to rm -rf local for a full rebuild.

[saraedum]

Replying to tkluck:

maybe .git is best. They contain data about branches, right? I would also expect to be able to rm -rf local for a full rebuild.

You are right. Does git care if there are extra files in .git?

[saraedum]

Replying to saraedum:

Replying to tkluck:

maybe .git is best. They contain data about branches, right? I would also expect to be able to rm -rf local for a full rebuild.

You are right. Does git care if there are extra files in .git?

No it does not. man git says:

Higher level SCMs may provide and manage additional information in the $GIT_DIR.

[ohanar]

Does not merge cleanly with build_system again, although easy fix. (More I want to hijack the branch here to review #15133)

[git]

Branch pushed to git repo; I updated commit sha1. Recent commits:

5adc4a4	Store the data of the SavingDicts? of SageDev? in .git
957a415	Small fix to legacy support for SavingDicts? in SageDev?
8b0d31e	Fixed a doctest in trac_interface.py
b8dccf1	SavingDicts? for SageDev? are now stored in the working tree
c66bdfd	fixed two problems in SageDev?.commit()

[git]

Branch pushed to git repo; I updated commit sha1. Recent commits:

a9c8a91	Merge branch 'build_system' into dev_scripts
5adc4a4	Store the data of the SavingDicts? of SageDev? in .git
957a415	Small fix to legacy support for SavingDicts? in SageDev?
8b0d31e	Fixed a doctest in trac_interface.py
b8dccf1	SavingDicts? for SageDev? are now stored in the working tree

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:7b0a041]	Fall back to http if ssl is not available
[changeset:07dfcef]	Be more verbose about whether a commit has been created or not in sagedev
[changeset:b8c78f2]	Hide git debug messages per default in sagedev.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:debc6ed]	Added a doctest to sagedev which shows how to start afresh when working on a ticket.
[changeset:c68940a]	Delete branch -> ticket association when abandon()ing a branch in sagedev.
[changeset:bc5ce5a]	Moved much of the code of create_ticket() to switch_ticket()
[changeset:2167665]	Make user interface for sagedev more tolerant.
[changeset:859987e]	Forbid None, True, False as branch names

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:ed8ed9d]	Fixed an infinite recursion in download_patch().
[changeset:0f30605]	Fixed a doctest for download_patch()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:4275bbd]	Fixed doctests if no user.name and user.email had been set in git
[changeset:e078628]	Add a warning in upload() to not erase dependencies.
[changeset:51e2179]	merge() fails when merging a ticket into itself

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:6b64c54]

switch_ticket()/create_ticket() works with an unclean working directory in some cases.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:8052e27]	Make sure we do not touch files outside of sage in the dev scripts.
[changeset:f43d853]	Corrected the _src directory in the git interface

[vbraun]

    sage: alice.git.echo.log('--pretty=%s') 
    Merge branch 'u/bob/ticket/1' of /dev/shm/... into ticket/1 

most peopole don't have their temp dir on /dev/shm. Can you ellipsize some more?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:76b09f0]	Make download_patch and import_patch download and apply patches like the patchbot does.
[changeset:73410f3]	Fixed two typos in docstrings/comments
[changeset:6061772]	fixed a TracConnectionError?() call
[changeset:f95ece6]	Fixed two optional: internet tests

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:167fe88]

Moved hg patch related methods to patch.py

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:f63d614]

Removed a reference to /dev/shm/ from sagedev.py

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:a625996]

Fixed a typo in import_patch()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:03cd3fd]	Fixed import_patch()
[changeset:9e3a961]	Fixed a question in import_patch()
[changeset:0b04731]	Fixed a typo in import_patch()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:ae2fde5]

Fixed edit_ticket().

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:51c04ac]	Use --work-tree in git_interface.
[changeset:a36c492]	Fixed an optional internet doctest in sagedev.
[changeset:4828a9a]	Revert "Make sure we do not touch files outside of sage in the dev scripts."

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:fa96127]	Delete dependencies when abandoning a ticket.
[changeset:736295c]	Stashes were printed twice by unstash()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:e793b41]	Implemented 'dependencies' in merge()
[changeset:9569608]	Several trivial doctest fixes in sage dev scripts.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:348e0a0]

"dependencies" is not a valid name for a branch since this is a special keyword for merge().

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:ad04f8d]	Fixing some problems in unstash
[changeset:98c0f6c]	Bugfixes to import_patch

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:c2902bb]	Improve diff() against 'dependencies'
[changeset:9e4a92c]	Added _is_master_uptodate() to dev scripts

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:37c477a]

Fixed a help message for upload().

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:bc1b755]

Improved local_tickets() to show the current branch and summary of the local tickets.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:6a43499]	Do not show summaries per default in local_tickets() because this might be very slow for some.
[changeset:6a38888]	make vanilla() return to the master branch in the dev scripts
[changeset:88ba93b]	Display param=True parameters correctly in help string for sagedev.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:665665c]

dev scripts upload ssh key only in upload()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:5979b18]

download_patch() added one level of tuple too many in a recursive call

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:bf35fac]

Minor change to diff() in dev scripts.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:93f5b8f]

Fixed a help string in create_ticket()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:fb05bb5]	Speed up local_tickets() by caching trac data
[changeset:38d12cd]	Use ​git://trac.sagemath.org/sage.git for unauthenticated access to the repository.

[vbraun]

How difficult would it be to preserve the description part of the u/user/description branch name? That is, when checking out a branch from a ticket #12345 use the local name ticket/12345/description? I think it'll be a bit annoying that the dev script users bulldoze your carefully thought-out description.

If the description already starts with ticket/12345/... then that first bit would be skipped. So if you push back to trac, it would become u/user/ticket/12345/description. And if somebody checks that ticket out, it would again become the local branch ticket/12345/description.

[saraedum]

That's not hard to do. I guess I would like to prompt the user because personally I prefer the short names.

Would you mind creating an issue at the github page?

[vbraun]

Do you want to spit in the original author's face and delete his description [Y/N]?

If you are using the dev scripts then you never get to see the actual branch name, so it shouldn't matter. If not then you should have tab-completion for git branches, making it into a non-issue as well.

In any case, we can easily add that later so its not important for the first release of the dev scripts.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:a82b16a]	Fixed an error in push()
[changeset:84de259]	Renamed commands in docstring in sagedev.py
[changeset:945e3cc]	Made create_ticket() not switch to branch for the ticket automatically.
[changeset:cedbf5c]	Renamed several commands in the dev scripts.
[changeset:003666a]	Made the sage-dev script more robust.

[vbraun]

I'm pretty happy with the rename. Two things remain:

    reset-to-clean-working-directory
                        Drop any uncommitted changes in the working directory.

should be something like

    clean               Delete all uncommited changes.

And unstash is not a word, its akin to "unspend" not being the opposite of "spend". It should be removed (my favorite) or renamed to `stash apply' (preferably) or 'stash-apply' (if we really must).

[roed]

Replying to vbraun:

I'm pretty happy with the rename. Two things remain:

    reset-to-clean-working-directory
                        Drop any uncommitted changes in the working directory.

should be something like

    clean               Delete all uncommited changes.

And unstash is not a word, its akin to "unspend" not being the opposite of "spend". It should be removed (my favorite) or renamed to `stash apply' (preferably) or 'stash-apply' (if we really must).

I'm happy with the clean rename, but I think unstash is better than either of the alternatives you've suggested, even if it's not a word.

[vbraun]

I always though unstash was supposed to be stash drop until I read the docs, found that very confusing.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:3a3fe00]	renamed reset_to_clean_working_directory() to clean() in the dev scripts
[changeset:3153282]	Fixed a help string in the dev scripts.

[vbraun]

I just debugged Simon's problem, and the dev scripts trip over the German localization in various places. The dev scripts should always set LC_ALL=C or posix before launching external programs if the output is being parsed. See also the thread at ​https://groups.google.com/d/msg/sage-git/jlm0piUE8Pk/DsA6lkbmhIsJ

[vbraun]

I tried out the "stash" feature and it doesn't stash anything. It reimplements something that is similar but using branches instead of the git stash stack. So if this is the desired effect then we shouldn't call it stash, which also resolves the unstash naming question. Just call its save/restore or something like that.

Also, documentation build failures:

[dev      ] /home/vbraun/Code/sage/src/doc/en/reference/dev/index.rst:2: WARNING: Title underline too short.
[dev      ] Sage's Development Scripts
[dev      ] =========
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/git_interface.py:docstring of sage.dev.git_interface.GitInterface.local_branches:5: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev.merge:60: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:11: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:10: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:11: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:11: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev.push:25: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev.show_dependencies:35: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/trac_interface.py:docstring of sage.dev.trac_interface.TicketSyntaxError:5: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/trac_interface.py:docstring of sage.dev.trac_interface.TicketSyntaxError:7: ERROR: Unexpected indentation.

[saraedum]

Replying to vbraun:

I tried out the "stash" feature and it doesn't stash anything. It reimplements something that is similar but using branches instead of the git stash stack. So if this is the desired effect then we shouldn't call it stash, which also resolves the unstash naming question. Just call its save/restore or something like that.

It implements something that feels like stashing but internally it does not stash, correct. The reason for this is that the names of stashes change, i.e., the first stash is called something like @stash{0} but if you create another stash then it changes to @stash{1}. Feel free to change the current behaviour/naming if you are not happy with it.

Also, documentation build failures:

[dev      ] /home/vbraun/Code/sage/src/doc/en/reference/dev/index.rst:2: WARNING: Title underline too short.
[dev      ] Sage's Development Scripts
[dev      ] =========
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/git_interface.py:docstring of sage.dev.git_interface.GitInterface.local_branches:5: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev.merge:60: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:11: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:10: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:11: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev:3: WARNING: Bullet list ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev:11: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev.push:25: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/sagedev.py:docstring of sage.dev.sagedev.SageDev.show_dependencies:35: WARNING: Literal block expected; none found.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/trac_interface.py:docstring of sage.dev.trac_interface.TicketSyntaxError:5: WARNING: Block quote ends without a blank line; unexpected unindent.
[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/trac_interface.py:docstring of sage.dev.trac_interface.TicketSyntaxError:7: ERROR: Unexpected indentation.

Thanks. I forgot to build the docs. I'll fix these later.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:0e56e5f]

Fixed doc warnings in the dev scripts.

[saraedum]

I can't make any sense of this warning:

[dev      ] /home/vbraun/Code/sage/local/lib/python2.7/site-packages/sage/dev/trac_interface.py:docstring of sage.dev.trac_interface.TicketSyntaxError:5: WARNING: Block quote ends without a blank line; unexpected unindent.

Could someone else have a look at this?

[vbraun]

All external programs ought to be called via sage-native-execute. Otherwise this is bound to happen (Fedora 19):

$ sage -dev create-ticket
nano: symbol lookup error: /lib64/libncursesw.so.5: undefined symbol: _nc_putchar
TicketSyntaxError: Editor returned non-zero exit value

[vbraun]

I fixed the doc build issues. Will try to find some time this week to go through the code.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:35d65d0]

fix doctest errors in dev scripts

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:efa4827]	Use quotation mark for quotation instead of backticks, misc code cleanup.
[changeset:91f05f4]	fix how external programs are called

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:c4a6bb4]

turn the patch.py module into a proper mixin class

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:95f4b83]

reflow UI text, ansi color support

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:2c6b1c8]	simplified the "sage -dev clean" command ui
[changeset:59aa09a]	removed the fake-stash-as-branches feature
[changeset:82169ab]	code cleanup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:f06fb16]	moving debug messages to UI.debug
[changeset:d5e58f7]	more sphinx->ascii for the sage-dev command line help

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:ace1bce]	textwrap a single "paragraph" at a time
[changeset:e2345ca]	make git.local_branches actually return branches sorted by commit time

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:b704805]

fix a doctests

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:1333f35]

small cleanup of sage-dev script

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:41878a2]

use the SageHelpFormatter? for subparsers

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:9bab11f]	fix fallback values for terminal dimensions
[changeset:e6f51c0]	sage-dev: decrement terminal width by 2 like parent class

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:5ddceb4]	some small fixes/cleanup to the pre-commit hook
[changeset:9ea5a2d]	refactor CmdLineInterface?.info (which also makes info respect the log level again)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:b55e0e6]

derp: really refactor CmdLineInterface?.info

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:98758d8]

default log level should probably be NORMAL

[vbraun]

I used INFO to log messages about how to proceed, this is one of the crucial features that make git usable:

$ git status
# On branch dev_scripts
# Your branch is ahead of 'trac/u/vbraun/dev_scripts' by 11 commits.
#   (use "git push" to publish your local commits)
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	test
nothing added to commit but untracked files present (use "git add" to track)

Everything with comment markers is what I call INFO. I spent a lot of time observing people trying to use the dev scripts last week and not being told about the next logical step is a huge obstacle.

There are lots of DEBUG messages (=stuff that protocols the execution flow like "operation completed successfully") that are mislabeled as INFO. If you don't want to see unless you are trying to debug the program then it should be DEBUG.

We can have an intermediate INFO < HELP < NORMAL level if somebody really has a use for non-debug informational messages but I think that would be overkill.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:e42d613]

fix doctests outside of sagedev.py

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:4063373]

more info messages, simplified user interface message formatting

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:c9dcab2]	more informational messages
[changeset:fc653db]	rename git.reset_to_clean_workspace_directory to git.clean_wrapper

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:934857e]	more code cleanups, minor UI rephrasing
[changeset:b558576]	renamed local-tickets -> tickets, prune-closed-tickets -> prune-tickets, misc fixes

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:8d8aeb7]

better display for SageDevValueError?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:39d7a1f]	fixed the optional - internet doctests
[changeset:11a8bf4]	cleaning up temp file usage

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:e136966]

clean up temp file usage (always use SAGE_TMP)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:d2efc20]

ensure that ticket_cache does not use Sage integer and improve error handling

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:5d16480]

fix minor issue in the sage -dev help

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:e439e14]

Always intercept git stdout/stderr in the dev scripts

[vbraun]

The last commit should fix the issue that Robert noted on ​https://groups.google.com/d/msg/sage-git/1V391vUBTH0/9qXjxD5u23UJ

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:08a3802]

guide user to merge, remove gather

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:45b0ecd]

lazy import dev scripts

[ohanar]

There is a non-trivial merge with this branch and 5.13.beta1 in src/bin/sage-fixdoctests (which I'm not familiar with). I've merge the latest beta into my build system branch, but not into master due to this conflict, please resolve.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

[changeset:9bd940c]	resolve conflict with newest build_system
[changeset:ec4ed11]	[FIXUP] 5.13.beta1: update git specific files
[changeset:a23df3e]	Merge branch 'upstream' into build_system
[changeset:b7dc123]	Merge 'sage-5.13.beta1'
[changeset:8693e80]	5.13.beta1
[changeset:755697a]	5.13.beta1
[changeset:e3808e5]	Trac #13726: Semimonomial transformation group
[changeset:d182ea9]	Trac #15279: init of RootSystem? corrected
[changeset:f5adfd5]	Solaris / ZFS fix
[changeset:d13cee2]	Trac #15270: Do not give up if the upstream shared library build fails

[vbraun]

I had made a quick&dirty fix to sage-fixdoctests since I needed it to fix the doctests... but in the meantime the proper fix from #10589 hit.

[vbraun]

The dev scripts fail on a clean install:

$ ./sage -dev
Traceback (most recent call last):
  File "/home/vbraun/opt/sage-git/sage-5.13.beta1/src/bin/sage-dev", line 9, in <module>
    from sage.dev import dev as DEV
ImportError: cannot import name dev

I haven't noticed before because this is masked by a stale .pyc file if you have been following development for a while. Fix in the following commit.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:55ef287]

fixed import statement for the commandline interface

[vbraun]

New commits:

[changeset:8380f43]

fixed missleading error message for trac authentication error

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:3af4a08]

remove , from my name

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:9f448b8]	don't abort ticket editing on nop push
[changeset:fdb6cdc]	remove last reference to download command
[changeset:ccac24d]	fix dev scripts in emacs shell

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:f835534]	dumb down pull to fix it
[changeset:d197b94]	Merge branch 'master' into dev_scripts

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:78cf9f9]

remove unused garbage

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:9e28043]

remove (now) bad uses of SageDev?.pull

[mderickx]

I have the impression that this ticket is still heavily under developement and hence should not have a "needs review" status.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:b5c6b09]

fix doctest errors in SageDev?.push

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:9785409]

SageDev?: remove branch keyword from pull

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:6207777]

SageDev?: fix pull doctest

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3880a5a

patch.py: deal with hg's silly copy commands

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ebb203c

trac_interface: encode fields with utf-8 as well

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f5f0ea4

also remove untracked directories after an aborted import patch

[vbraun]

New commits:

​702d7af

do not suggest force push

[vbraun]

I agree with everything that I haven't written myself here... Andrew, is there anything left that we need to fix?

[ohanar]

Not that I know of.

[vbraun]

I get doctest failures on fresh accounts (cached stuff in DOT_SAGE?) of the type

File "src/sage/dev/git_interface.py", line 118, in sage.dev.git_interface.GitProxy._run_git
Failed example:
    dev.git.status()
Expected:
    Traceback (most recent call last):
    ...
    AssertionError: working with the sage repository in a doctest
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 480, in _run
        self.execute(example, compiled, test.globs)
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 839, in execute
        exec compiled in globs
      File "<doctest sage.dev.git_interface.GitProxy._run_git[7]>", line 1, in <module>
        dev.git.status()
      File "lazy_import.pyx", line 312, in sage.misc.lazy_import.LazyImport.__getattr__ (sage/misc/lazy_import.c:2402)
      File "lazy_import.pyx", line 248, in sage.misc.lazy_import.LazyImport._get_object (sage/misc/lazy_import.c:1885)
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/dev/sagedev_wrapper.py", line 250, in <module>
        dev = SageDevWrapper(SageDev())
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/dev/sagedev.py", line 119, in __init__
        self.trac = TracInterface(self.config['trac'], self._UI)
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/dev/trac_interface.py", line 212, in __init__
        self.__ticket_cache = SavingDict(ticket_cache_file)
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/dev/saving_dict.py", line 116, in __init__
        self._write()
      File "/home/buildslave/slave/local_full/build/local/lib/python2.7/site-packages/sage/dev/saving_dict.py", line 236, in _write
        assert os.path.abspath(self._filename).startswith(SAGE_TMP), error
    AssertionError: write attempt to a saving_dict in a doctest

Doesn't happen in my own user account. Anybody knows whats up?

[vbraun]

On skynet/mark (Sparc Solaris, Sun_SSH_1.1) you must specify the key type in ssh-keygen:

-bash-3.00$ ssh-keygen -q -f /tmp/foo -P ''
You must specify a key type (-t).
Usage: ssh-keygen [options]
Options:
  -b bits     Number of bits in the key to create.
  -c          Change comment in private and public key files.
  -e          Convert OpenSSH to IETF SECSH key file.
  -f filename Filename of the key file.
  -i          Convert IETF SECSH to OpenSSH key file.
  -l          Show fingerprint of key file.
  -p          Change passphrase of private key file.
  -q          Quiet.
  -y          Read private key file and print public key.
  -t type     Specify type of key to create.
  -B          Show bubblebabble digest of key file.
  -C comment  Provide new comment.
  -N phrase   Provide new passphrase.
  -P phrase   Provide old passphrase.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6a512d6

Explicitly specify the key type in ssh-keygen (requried for Sun SSH)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2941308

Use a DoctestSageDevWrapper? for the global dev object during doctests

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​946b5b4

fix the commandline sage -dev script

[tscrim]

I am having trouble importing Hg patches. I added an issue to the Github page with the details.

EDIT - Also I don't like that when aborting patch imports that it deletes all untracked directories (in particular, documentation output). I'd prefer if we could only delete newly created directories.

Thanks for all your work on this.

[vbraun]

Which github page? In any case, our bug tracker is not on github. I'm sure there are a lot of small things that could be improved, but improvement requests should go into a separate ticket by now. Unless you think that the deleting of documentation output merits delaying the git transition.

[tscrim]

Replying to vbraun:

Which github page? In any case, our bug tracker is not on github.

The issues forum in the ticket description: ​​https://github.com/sagemath/sage-workflow/issues

I'm sure there are a lot of small things that could be improved, but improvement requests should go into a separate ticket by now. Unless you think that the deleting of documentation output merits delaying the git transition.

It's not worth the delay. It's an annoyance IMO, but not horribly so.