Context Navigation

Back to Ticket #3905

Ticket #3905: prog.patch

File prog.patch, 160.2 KB (added by jhpalmieri, 5 years ago)

commontex/macros-new.tex

# HG changeset patch
# User J. H. Palmieri <palmieri@math.washington.edu>
# Date 1219199147 25200
# Node ID 8f9275ea4358f44711241bbc022128ef6a46d202
# Parent  bba79837cb391923027900edb059116505472ab7
revising programming guide

diff -r bba79837cb39 -r 8f9275ea4358 commontex/macros-new.tex

-                      a
 % when run through latex2html.  See the command \newbf above for an
 % application of this.  See the latex2html docs for more information.
 \usepackage{html}
+\usepackage{verbatim}
+% the following is used for nesting verbatim environments, used in prog/prog.tex
+\newenvironment{newverbatim}%
+  \verbatim%
+  {\endverbatim}
 \newcommand{\by}[1]{\par\noindent{\bf AUTHOR:} #1\par}

prog/prog.tex

diff -r bba79837cb39 -r 8f9275ea4358 prog/prog.tex

-                      a
+% see 3393, 3882
+% coercion section
+% mutability section should be rewritten, expanded.
+% benchmarking?
 \documentclass{manual}
 \usepackage{textcomp}
+\include{macros}
+\include{macros-new}
 % a hack to get around another hack.
 \ifpdf
 \usepackage{url}
 \else
 \fi
 %\newcommand{\todo}[1]{\footnote{1}}
 \newcommand{\todo}[1]{}
 \newcommand{\swlist}{GAP, GSL, Matplotlib, Maxima, MWRANK, NetworkX,
   NTL, Numpy, PARI and Singular}
+  NTL, Numpy, PARI and Singular\xspace}
 \usepackage{color}
 \definecolor{hue}{rgb}{.202, .602, .58}
 …
 \begin{document}
 \title{\SAGE Programming Guide}
+\title{\SAGE Developer's Guide}
+\author{William Stein%\thanks{Associate Professor, University of Washington, wstein@math.washington.edu.}
+\ \ and David Joyner%\thanks{Mathematics Department, US Naval Academy, wdj@usna.edu}
+}
+\author{William Stein and David Joyner}
 \maketitle
 …
 \begin{abstract}
 \noindent
+Absolutely {\bf everybody} who uses \SAGE should contribute something
+\SAGE is a free mathematics software system. It is implemented using
+Python, Cython, and C++, and uses \swlist.  It is free and open
+source, and is available under the terms of the GNU Public License.
+Everybody who uses \SAGE should contribute something
 back to \SAGE at some point.  Implement a new function, add examples
 to the documentation, find bugs and typos, fix a bug, create a new
 class, create a fast new C library, etc.  This book is a guide about
 how to contribute code back to \SAGE.
+how to contribute to \SAGE.
+This document describes how to write programs using \SAGE, and also
+how to modify and extend the core \SAGE libraries.
+\SAGE stands for Software for Algebra and Geometry Experimentation.
+It is implemented using Python, Cython, and C++, and uses \swlist.
+It is free and open source, and is available under the terms of the
+GNU Public License.
+This document describes how to write programs using \SAGE,
+how to modify and extend the core \SAGE libraries, and how to modify
+\sage's documentation.  It also discusses how to share your new and
+modified code with other \SAGE users.
 \end{abstract}
 \tableofcontents
+\todo{Remove I and me}
+\part{Writing Code for \sage}
 \chapter{Introduction}
 If there is something you would like to implement and make
 …
 contains sophisticated, optimized number theory algorithms.  Notably
 absent from this triad is a good system for exact linear algebra
 (something MAGMA does extremely well), but this gap is being filled by
+code being written for \SAGE. \todo{Say something about GSL,
+  Matplotlib, Maxima, MWRANK, NetworkX, NTL, Numpy}
+%Unfortunately, Singular, GAP, and PARI
+%have vastly different languages and conventions.
+code written for \SAGE.
+\todo{Say something about GSL, Matplotlib, Maxima, MWRANK, NetworkX,
+NTL, Numpy}
 \SAGE is not just about gathering together functionality (something
 UNIX already does well), it is about providing a clear, systematic and
+\SAGE is not just about gathering together functionality, it is about
+providing a clear, systematic and
 consistent way to access a large number of algorithms, in a coherent
 framework that makes sense mathematically.  In the design of \SAGE, the
 semantics of objects, the definitions, etc., are informed by how the
 …
 This document was authored by William Stein, David Joyner and others
 with the editorial help of Iftikhar Burhanuddin.
-\chapter{\SAGE{} Coding Conventions}\label{ch:conventions}
-To meet the goal of making \sage easy to read, all Python/Cython code
-that is included with \sage should adhere to the style conventions
-mentioned in this chapter.
+\section{Coding Conventions}
+Follow the standard Python formatting rules when writing code for Sage,
+as explained at \url{http://www.python.org/doc/essays/styleguide.html}.
+\chapter{Conventions for Coding in \SAGE}\label{ch:conventions}
+To meet the goal of making \sage easy to read, maintain, and improve,
+all Python/Cython code that is included with \sage should adhere to
+the style conventions discussed in this chapter.
+\section{Python Coding Conventions}
+Follow the standard Python formatting rules when writing code for \sage,
+as explained at
+\url{http://www.python.org/dev/peps/pep-0008/} and
+\url{http://www.python.org/dev/peps/pep-0257/}.
 In particular,
 \begin{itemize}
 \item Use 4 spaces for indentation levels.  Do not use tabs as they
 …
 \begin{verbatim}
         def set_some_value()
 \end{verbatim}
      instead of:
+     instead of CamelCase:
 \begin{verbatim}
         def SetSomeValue()
 \end{verbatim}
 …
 \end{itemize}
+\section{File and Directory names}
+\section{File and Directory Names}
 Python \sage library code uses the following conventions.  Directory
 names may be plural (e.g., \code{rings}) and file names are almost
 always singular (e.g., \code{polynomial\_ring.py}).  Note that the
 file \code{polynomial\_ring.py} might still contain definitions of
 several different types of polynomial rings.
 \note{You are strongly encouraged
+\note{You are encouraged
 to include miscellaneous notes, emails, design
 discussions, etc., in your package.  Make these
 plain text files (with extension \code{.txt})
+in a subdirectory directory called \code{notes}.
+in a subdirectory called \code{notes}.  (For example, see
+\code{SAGE_ROOT/devel/sage/sage/ext/notes/}.)
+}
-%\section{Ordering of function names in a class}
-%Function names
-%should be defined in alphabetical order, with \code{__}'s first,
-%then \code{_}'s, then public functions.   This is for easy of readability
-%and finding code.  It also encourages functions with related behavior
-%to be named in a similar way, which makes it easier to find them
-%in the documentation and when using tab completion from the interpreter.
 \section{Headings of \SAGE Library Code Files}
+The top of each \sage code file should appear as follows\todo{What is \section{Tutorial} below}:
+The top of each \sage code file should follows this format:
 \begin{verbatim}
 r"""
 …
 ...
 AUTHORS:
     - YOUR NAME (2005-01-03): initial version
     - person (date in ISO year-month-day format): short desc
+    -- YOUR NAME (2005-01-03): initial version
+    -- person (date in ISO year-month-day format): short desc
      ...
     - person (date in ISO year-month-day format): short desc
+    -- person (date in ISO year-month-day format): short desc
-\section{Tutorial}
  ...
 Lots and lots of examples.
 """
 #*****************************************************************************
+#       Copyright (C) 2006 William Stein <wstein@gmail.com>
+#                     2006 YOUR NAME <your email>
+#       Copyright (C) 2008 YOUR NAME <your email>
+#
 #  Distributed under the terms of the GNU General Public License (GPL)
 #                  http://www.gnu.org/licenses/
 #*****************************************************************************
 \end{verbatim}
 The following is top of the ssmod.py file, which contains the
+SupersingularModule code and is located at
 \code{SAGE_ROOT/devel/sage/sage/modular/ssmmod/ssmod.py}.
+The following is the top of the file
+\code{SAGE_ROOT/devel/sage/sage/modular/ssmmod/ssmod.py}, which
+contains the Supersingular Module code.
 {\small
 \begin{verbatim}
 """
 Module of Supersingular Points
 …
+    [
     (Vector space of degree 33 and dimension 1 over Finite Field of size 97
     Basis matrix:
+    [ 0  0  0  1 96 96  1  0 95  1  1  1  1 95  2 96  0  0 96 96  0  0 96
+96 96  2  0  0  1  1 95  0], 1),
+    [ 0  0  0  1 96 96  1 96 96  0  2 96 96  0  1  0  1  2 95  0  1  1  0  1  0 95  0 96 95  1 96  0  2], True),
     (Vector space of degree 33 and dimension 1 over Finite Field of size 97
     Basis matrix:
+    [ 0  1 96 16 75 22 81  0  0 17 17 80 80  0  0 16  1 40 74 23 57 96 81
+23 74  0  0  0 24 73  0  0], 1),
+    [ 0  1 96 75 16 81 22 17 17  0  0 80 80  1 16 40 74  0  0 96 81 23 57 74  0  0  0 24  0 23 73  0  0], True),
     (Vector space of degree 33 and dimension 1 over Finite Field of size 97
     Basis matrix:
+    [ 0  1 96 90 90  7  7  0  0  6 91  6 91  0  0  7 13  0 91  6  0 84 90
+91  6  0  0  0 90  7  0  0], 1)
+    [ 0  1 96 90 90  7  7  6 91  0  0 91  6 13  7  0 91  0  0 84 90  6  0  6  0  0  0 90  0 91  7  0  0], True)
+    ]
     sage: len(D)
 We compute a Hecker operator on a space of huge dimension!
     sage: X = SupersingularModule(next_prime(100000))
     sage: t = X.T(2).matrix()  # long time (but still less than a minute!)
     sage: t.nrows()            # long time
+    sage: t = X.T(2).matrix()            # long time (but still less than a minute!)
+    sage: t.nrows()                      # long time
+TESTS:
+    sage: X = SupersingularModule(389)
+    sage: T = X.T(2).matrix().change_ring(QQ)
+    sage: d = T.decomposition()
+    sage: len(d)
+    sage: [a[0].dimension() for a in d]
+    [1, 1, 2, 3, 6, 20]
+    sage: loads(dumps(X)) == X
+    True
+    sage: loads(dumps(d)) == d
+    True
 """
 #*****************************************************************************
 …
 }%small
 All code included with \sage must be licensed under the GPL or a less
 restrictive license (e.g., the BSD license).  You should share
 copyright with William Stein by including your name at the top.  It is {\em very
+restrictive license (e.g., the BSD license).
+It is {\em very
   important} that you include your name in the AUTHOR log, since
 everybody who submits code to \sage to receive proper credit.  (If
 ever you feel you are not receiving proper credit for anything you
 submit to \sage, please let William Stein know!)
+submit to \sage, please let the development team know!)
 \section{Documentation Strings}\label{sec:conv-docs}
+{\bf Every} function should have a docstring that includes
+\textbf{Every} function must have a docstring that includes
 the following information:
 \begin{itemize}
 \item A one-sentence description of the function, followed by a blank line.
 …
 \item An EXAMPLES block for examples. This is {\em not} optional.
 \item A NOTES block for special notes (optional). Include information
   such as algorithm used, references, etc.
+\item An AUTHORS block (important!).
+\item An AUTHORS block (optional, but encouraged for important
+functions, so users can see from the docstring who wrote it and
+therefore whom to contact if they have questions).
 \end{itemize}
 Use the following template when documenting functions. Note the
 indentation:
 %skip
 \begin{verbatim}
 def point(self, x=1, y=2):
     r"""           # note the r for "raw string"
+    r"""
     This function returns the point $(x^5,y)$.
     INPUT:
 …
 \begin{itemize}
 \item Use nice \Latex formating everywhere but in the
+  \code{INPUT/OUTPUT} blocks.
+  \code{INPUT/OUTPUT} blocks.  If you use backslashes, either use
+  double backslashes or place an {\tt r} right before the first triple
+  opening quote. For example,
+%skip
+\begin{verbatim}
+def cos(x):
+    """
+    Returns $\\cos(x)$.
+    """
+def sin(x):
+    r"""
+    Returns $\sin(x)$.
+    """
+\end{verbatim}
+Also, the non-standard \Latex macros that are used when typesetting the
+documentation are defined in
+\code{SAGE_ROOT/doc/commontex/macros.tex}.  If you need to add more
+macros, add them there and post a patch file on the \sage trac server
+(Chapter~\ref{ch:trac}) so that they can be included in the next
+version of \sage.
 \item Liberally describe what the examples do.  Note that there must
   be a blank line after the example code and before the explanatory
 …
   regular basis, and are crucial for the quality and adaptability of
   \sage.  Without such examples, small changes to one part of \sage
   that break something else might not go seen until much later when
   someone uses the system, which is frustrating.
+  someone uses the system, which is unacceptable.
 \end{itemize}
 The code in the examples should pass automatic testing.  This means
 that if the above code is in the file \code{f.py} (or \code{f.sage}),
 then \code{sage -t f.py} should not give any error messages.  Testing
 occurs with full \sage preparsing of input within the standard \sage
 shell environment.  {\em Important.}\todo{Clarify with an
   example/pathology and explanation. }  The file \code{f.py} is not
+shell environment, as described in Section~\ref{sec:Preparsing}.
+{\em Important:} The file \code{f.py} is not
 imported when running tests unless you have arranged that it be
 imported into your \sage environment, i.e., unless its functions are
+available when you start \sage using the \code{sage} command.
+available when you start \sage using the \code{sage} command.  For
+example, the function \code{cdd_convert} in the file
+\code{SAGE_ROOT/devel/sage/sage/geometry/polyhedra.py} includes
+an EXAMPLES block containing the following:
+\begin{verbatim}
+    sage: from sage.geometry.polyhedra import cdd_convert
+    sage: cdd_convert(' 1 1 0 0')
+    [1, 1, 0, 0]
+\end{verbatim}
+\sage doesn't know about the function \code{cdd_convert} by default,
+so it needs to be imported before it is tested; hence the first line
+in the example.
+\subsection{Further conventions for automated testing of examples}
+\subsection{Further Conventions for Automated Testing of Examples}
 \label{sec:Further-conventions}
 The Python script \code{local/bin/sage-doctest} implements
 documentation testing in \sage (see Chapter~\ref{ch:testing} for more
+documentation testing in \sage (see Section~\ref{ch:testing} for more
 details).  When writing documentation, keep the following points in
 mind:
 \begin{itemize}
 …
   \code{2/3} as a rational instead of the Python int \code{0}. For
   more information on preparsing see Section \ref{sec:Preparsing}.
 \item If a test line contains the text \code{random} it is executed by
+\item If a test line contains the text \code{random}, it is executed by
   \code{sage-doctest} but \code{sage-doctest} does not check that the
   output agrees with the output in the documentation string.
+  For example, the docstring for the \code{__hash__} method for
+  \code{CombinatorialObject} in
+  \code{SAGE_ROOT/devel/sage/sage/combinat/combinat.py} includes the
+  lines
+%skip
+\begin{verbatim}
+    sage: hash(c) #random
+    1335416675971793195
+    sage: c._hash #random
+    1335416675971793195
+\end{verbatim}
 \item If a line contains the text \code{long time} then that line is
   not tested unless the {\code -long} option is given, e.g.,
 …
   \sage development, but will get run before major releases.  No
   example should take more than about 30 seconds.
   Here is the \code{Lseries} function docstring which contains the phrase
   \code{long time} and is taken from
+\code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_rational.py}
+  For example, here is part of the docstring from the \code{regulator} method for
+  rational elliptic curves, from the file
+  \code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_rational.py}:
+%skip
 \begin{verbatim}
+    def Lseries(self, s):
+        r"""
+        Returns the value of the L-series of the elliptic curve E at s, where s
+        must be a real number.
+        Use self.Lseries_extended for s complex.
+        \note{If the conductor of the curve is large, say $>10^{12}$,
+        then this function will take a very long time, since it uses
+        an $O(\sqrt{N})$ algorithm.}
+        EXAMPLES:
+            sage: E = EllipticCurve([1,2,3,4,5])
+            sage: E.Lseries(1)
+.000000000000000
+            sage: E.Lseries('1.1')       # long time (!)
+.28549100767814833
+            sage: E = EllipticCurve([0, 0, 1, -1, 0])
+            sage: E.regulator()              # long time (1 second)
+.0511114082399688
 \end{verbatim}
 \item If a line contains \code{todo: not implemented}, it is never
 tested.   It is good to include lines like this to make clear
 …
 \code{sage -t -optional f.py}.  (Note that \code{-optional}
 must not be the first argument to \code{sage}.)  Use this
 to include doctests that require optional packages.
+For example,
+For example, the docstring for \code{_magma_init_} in the
+class \code{EllipticCurve_finite_field} from the
+\code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_finite_field.py}
+contains
 \begin{verbatim}
     sage: E.padic_regulator(5)  # requires optional MAGMA package
     xxx
+    sage: EllipticCurve(GF(41),[2,5])._magma_init_() # optional -- requires Magma
+    'EllipticCurve([_sage_[1]|GF(41)!0,GF(41)!0,GF(41)!0,GF(41)!2,GF(41)!5])'
 \end{verbatim}
 \item If the entire documentation string contains all three words
 …
 \end{itemize}
+Using \code{sage_search} from the \sage prompt or \code{grep} one can
+easily find aforementioned keywords and in the case of \code{todo: not
+  implemented} use the results to motivate further development on \sage.
+Using \code{search_src} from the \sage prompt (or \code{grep}), one can
+easily find the aforementioned keywords.  In the case of \code{todo: not
+ implemented}, one can use the results of such a search to direct
+further development on \sage.
+\section{Exceptions}
+Whenever code such as the following should be avoided:
+\section{Automated Testing}\label{ch:testing}
+This section describes \sage's automated testing of test files of the
+following types: \code{.py .pyx .sage .tex}.  Briefly, use \code{sage -t
+  <file>} to test that the examples in \code{<file>} behave exactly as
+claimed.  See the following subsections for more details.
+See also Section~\ref{sec:conv-docs} for a discussion of how to
+include examples in documentation strings and what conventions to
+follow.
+\subsection{Testing .py, .pyx and .sage Files}\label{sec:test:python}
+Run \code{sage -t <filename.py>} to test that all code examples in
+\code{filename.py}.  Similar remarks apply to \code{.sage} and
+\code{.pyx} files.
+\begin{verbatim}
+  sage -t [--verbose] [--optional]  [files and directories ... ]
+\end{verbatim}
+When you run \code{sage -t <filename.py>}, \sage makes a copy of
+\code{<filename.py>} with all the \code{sage} prompts replaced by
+\code{{>}{>}>}, then uses the standard Python doctest framework to test
+the documentation.  More precisely,
+the Python script \code{local/bin/sage-doctest} implements
+documentation testing.  It does the following when asked
+to test a file \code{foo.py} or \code{foo.sage}.
+\begin{enumerate}
+\item Creates the directory \code{.doctest} if it doesn't
+exist and the file \code{.doctest/foo.py}.
+\item The file \code{.doctest_foo.py} contains functions for
+each docstring in \code{foo.py}, but with all \sage preparsing
+applied and with \code{from sage.all import *} at the top.
+The function documentation is thus standard Python with
+\code{{>}{>}>} prompts.
+\item \code{sage-doctest} then runs \sage's Python interpreter
+on \code{.doctest_foo.py}.
+\end{enumerate}
+\note{{\em Note that line numbers in the errors
+you might see apply to the file \code{.doctest_foo.py}, not
+to the original file \code{foo.py}!}  If you get errors,
+feel free to inspect \code{.doctest_foo.py}; it is never
+automatically deleted by \sage.}
+Your file passes these tests if the code in it will run when entered
+at the \code{sage:} prompt with no special imports.  Thus users are
+guaranteed to be able to exactly copy code out of the examples you
+write for the documentation and have them work.
+\subsection{Testing \Latex Documentation}
+Run \code{sage -t <filename.tex>} to test the examples in
+verbatim environments in \Latex documentation.   Put
+\code{\%skip}
+on the line right before a verbatim environment to have that
+example skipped when testing the file.
+\sage creates a file \code{.doctest_filename.py} and tests
+it just as for \code{.py}, \code{.pyx} and \code{.sage}
+files.   In order to skip testing of a block of code
+in a verbatim environment, put the \Latex comment
+\code{\%skip} on the previous line.
+Of course in \Latex files, one
+often inserts explanatory texts between different
+verbatim environments.    To link together verbatim
+environments use the \code{\%link} comment.  For
+example:
 %skip
+\begin{verbatim}
+try:
+    some code
+except:               # bad
+    more code
+\end{verbatim}
+Instead catch specific exceptions. For example,
+\begin{verbatim}
+try:
+    return self.__coordinate_ring
+except (AttributeError, other exceptions), msg:           # Good
+    more code to compute something
+\end{verbatim}
+Note that the syntax in the except is to list all the exceptions
+that are caught as a tuple, followed by the variable that holds
+the message.
+\begin{newverbatim}
+    \begin{verbatim}
+    sage: a = 1
+    \end{verbatim}%link
+If you don't have any exceptions explicitly listed (as a tuple), your
+code will catch absolutely anything, including ctrl-C and alarms,
+and this will lead to confusion.
+    Next we add 1 to \code{a}.
+    %link
+    \begin{verbatim}
+    sage: 1 + a
+    \end{verbatim}
+\end{newverbatim}
+\section{Optional Packages} If an optional package is required for a
+certain function to work, that function should fail gracefully ---
+perhaps using a \code{try} and \code{except} block --- when the optional
+package is not available, and should give a hint about how to install
+it.  For example, typing \code{sage -optional} gives a list of all
+optional packages, so suggest to the user that they type that.
+See \code{SAGE_ROOT/doc/tut/tut.tex} for many examples
+of how to include automated testing in \Latex documentation
+for \sage.
+\section{Randomized Testing}
+\label{ch:randomtesting}
+In addition to all the examples in your docstrings, which serve as
+both demonstrations and tests of your code, you should consider creating a
+test suite.  Think of this as a program that will run for a while and
+``tries'' to crash your code using randomly generated input.  Your
+test code should define a class \class{Test} with a {\tt random()}
+method that runs random tests.  These are all assembled together
+later, and each test is run for a certain amount of time on a regular
+basis.
+For example, see the file \code{SAGE_ROOT/devel/sage/sage/modular/modsym/tests.py}.
+\chapter{Coding in Python for \SAGE}
+This chapter discusses some issues with, and advice for, coding in \SAGE.
+\section{Design}
+If you are planning to develop some new code for \sage, design is
+important.  So think about what your program will do and how that fits
+into the structure of \SAGE.  In particular, much of \SAGE is
+implemented in the object-oriented language Python, and there is a
+hierarchy of classes that organize code and functionality.  For
+example, if you implement elements of a ring your class should derive
+from \class{sage.structure.element.RingElement}, rather than starting
+from scratch.  Try to figure out how your code should fit in with
+other \sage code, and design it accordingly.
 \section{Special \sage Functions}
 …
 also has special methods.  They are typically of the form
 \code{_XXX_}.  (In a few cases the trailing underscore is not
 included, but this will be changed so that the trailing underscore is
 always included.)  This section describes all special methods.
+always included.)  This section describes these special methods.
 All objects in \sage should derive from the Cython extension
 class \code{SageObject}:
 …
 class MyClass(SageObject,...):
     ...
 \end{verbatim}
+or from some other already existing \sage class:
+%skip
+\begin{verbatim}
+from sage.rings.ring import Algebra
+class MyFavoriteAlgebra(Algebra):
+    ...
+\end{verbatim}
 You should implement the \code{\_latex\_} and \code{\_repr\_} method
 for every object. The other methods depend on the nature of the
 …
 \subsection{\Latex Representation}
 Every object x in \sage should support the command \code{latex(x)}, so
 that any \sage object can be easily and accurately displayed via
 \Latex.  Here is how to make a class and therefore its instances
+\Latex.  Here is how to make a class (and therefore its instances)
 support the command \code{latex}.
 \begin{enumerate}
+\item Put \code{import sage.misc.latex as latex} at the top of the
+  file that defines your class.
+\item Define a member function \code{_latex_(self)} that returns a
+\item Define a method \code{_latex_(self)} that returns a
   \Latex representation of your object.  It should be something that
   can be typeset correctly within math mode.
+\item Often objects are built up out of other options.  For example if
+\item Often objects are built up out of other \sage objects, and these
+  components should be typeset using the \code{latex} function.
+  For example if
   \code{c} is a coefficient of your object, and you want to typeset
   \code{c} using latex, use \code{latex(c)} instead of
   \code{c._latex_()}, since \code{c} might not have a \code{_latex_}
 …
   illustrates latex generation for your object.
 \item You can use any macros included in \code{amsmath},
   \code{amssymb}, \code{amsfonts} or the ones defined in
+  \code{amssymb}, or \code{amsfonts}, or the ones defined in
   \code{SAGE_ROOT/doc/commontex/macros.tex}.
-\item Use \code{view(x)} to view the typeset version of an object x.
 \end{enumerate}
 %skip
 An example template for \Latex representation follows:
+An example template for a \code{_latex_} method follows:
 \begin{verbatim}
-import sage.misc.latex as latex
-...
 class X:
    ...
    def _latex_(self):
 …
        return `\\frac{%s}{%s}''%(latex(self.numer), latex(self.denom))
 \end{verbatim}
+As shown in the example, \code{latex(a)} will produce \Latex code
+representing the object \code{a}.  Calling \code{view(a)} will display the
+typeset version of this.
 \subsection{Print Representation}
 The standard Python printing method is \code{__repr__(self)}.  In
 \sage, that is for objects that derive from \code{SageObject} (which
 is everything in \sage), instead define \code{_repr_(self)}.  This is
+preferable because if you only define \code{_repr_(self)} and not
 \code{__repr__(self)}, then users can rename your object to print
 however they like.
+is everything in \sage), instead define \code{_repr_(self)}.
+% This is preferable because if you only define \code{_repr_(self)} and
+% not \code{__repr__(self)}, then users can rename your object to print
+% however they like.
 Here is an example of the \code{_latex_} and \code{_repr_} functions
 for the Pi class. It is from the file
 \code{SAGE_ROOT/devel/sage/sage/functions/constants.py}
+\code{SAGE_ROOT/devel/sage/sage/functions/constants.py}:
+\begin{verbatim}%skip
+%skip
+\begin{verbatim}
 class Pi(Constant):
     """
     The ratio of a circle's circumference to its diameter.
 …
         return "\\pi"
 \end{verbatim}
 \subsection{Matrix from Object}
 Provide a \code{_matrix_} member function for an object that can be
+\subsection{Matrix or Vector from Object}
+Provide a \code{_matrix_} method for an object that can be
 coerced to a matrix over a ring $R$. Then the \sage function
 \code{matrix} will work for this object.
 The following is from \code{SAGE_ROOT/devel/sage/sage/graphs/graph.py}
+The following is from \code{SAGE_ROOT/devel/sage/sage/graphs/graph.py}:
 \begin{verbatim}
+class SimpleGraph(GenericGraph):
+   ...
+   def _matrix_(self, R):
+       return self.am()
+class GenericGraph(SageObject):
+    ...
+    def _matrix_(self, R=None):
+        if R is None:
+            return self.am()
+        else:
+            return self.am().change_ring(R)
+   def am(self):
        """
        Shorter call for adjacency matrix makes life easier.
+       """
        return self.adjacency_matrix()
+    def adjacency_matrix(self, sparse=None, boundary_first=False):
+        ...
+    am = adjacency_matrix # shorter call makes life easier
 \end{verbatim}
+\subsection{Vector from Object}
 Provide a \code{_vector_} member function for an object that can be
+Similarly,
+provide a \code{_vector_} method for an object that can be
 coerced to a vector over a ring $R$. Then the \sage function
 \code{vector} will be work for this object.
+\code{vector} will work for this object.
 \todo{Provide example from a .py file
   The following is from the file
   SAGE_ROOT/sage/sage/modules/free_module_element.pyx
+  \code{SAGE_ROOT/sage/sage/modules/free_module_element.pyx}:
 \begin{verbatim}
 cdef class FreeModuleElement(element_Vector):   # abstract base class
 …
+}
+\section{\SAGE Preparsing}
+\label{sec:Preparsing}
+The following files are relevant to preparsing in \sage:
+\begin{enumerate}
+\item \code{SAGE_ROOT/local/bin/sage-sage},
+\item \code{SAGE_ROOT/local/bin/sage-preparse},
+\item \code{SAGE_ROOT/devel/sage/sage/misc/preparser.py}
+% \item \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py}
+\todo{Talk about
+    \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py} file}
+\end{enumerate}
+In particular, the file \code{preparser.py} contains the \SAGE
+preparser code.  Here are some , and the following are some notes from it:
+\begin{itemize}
+\item In \SAGE, methods can be called on integer and real literals --
+  note that in pure Python this would be a syntax error.  For example:
+\begin{verbatim}
+    sage: 16.sqrt()
+    sage: 87.factor()
+* 29
+\end{verbatim}
+\item Raw literals are not preparsed, which can be useful from an
+  efficiency point of view.  Just like Python ints are denoted by an
+  L, in \SAGE raw integer and floating literals are followed by an ``r''
+  (or ``R'') for raw, meaning not preparsed.  Example:
+\begin{verbatim}
+    sage: a = 393939r
+    sage: a
+    sage: type(a)
+    <type 'int'>
+    sage: b = 393939
+    sage: type(b)
+    <type 'sage.rings.integer.Integer'>
+    sage: a == b
+    True
+\end{verbatim}
+\item Raw literals can be very useful in certain cases. For instance,
+  Python integers are typically much more efficient than \SAGE integers
+  when they are very small; large \SAGE integers are much more
+  efficient than Python integers, since they are implemented using the
+  GMP C library.  For example, the first of these commands may be somewhat
+  slower than the second:
+\begin{verbatim}
+    sage: v = [ 2*i for i in range(10000)]
+    sage: v = [ 2r*i for i in range(10000r)]
+\end{verbatim}
+because the first uses \sage integers, while the second uses raw
+Python integers.
+\end{itemize}
+Consult the file \code{preparser.py} for more details about \SAGE
+preparsing, more examples involving raw literals, etc.
+When a file \code{foo.sage} is loaded in a \sage session, a preparsed version of
+\code{foo.sage} is created titled \code{foo.py}. The beginning of \code{foo.py} states:
+\verb+This file was *autogenerated* from the file foo.sage.+
 \section{The \SAGE Rules and Conventions for Coercion and Arithmetic}
-\note{These are the {\bf planned} rules and conventions.  They have not
-been systematically implemented throughout the system yet.}
+Let $R$ and $S$ be \SAGE parent structures, e.g.,
+groups, rings, point sets, etc.\todo{Make section gentler}
+\note{August 2008: Much of this material is out of date.  We are
+working on a revised version.}
+\todo{Make section gentler}
+Let $R$ and $S$ be \SAGE parent structures, e.g., groups, rings, point
+sets, etc.
 \begin{itemize}
 \item {\bf OBJECT COERCION \code{_coerce_}:}
+\item \textbf{OBJECT COERCION \code{_coerce_}:}
  Suppose a \code{_coerce_} map $R \to S$ is defined.  This
 is a method of $S$, so it is called using the notation \code{S._coerce_(x)}, where $R$ is a parent of $x$.
 It's signature is
 …
 rules.  E.g., embeddings of finite fields via Conway polynomials, or
 inclusions of extensions of number fields, fit into this structure, as
 does the inclusion $\QQ \hookrightarrow \CC$ and the surjection $\ZZ
 \to \ZZ/n\ZZ$.  The function \code{_coerce_} {\bf does not} have to be
+\to \ZZ/n\ZZ$.  The function \code{_coerce_} \textbf{does not} have to be
 ``canonical'' in a precisely defined mathematical sense.
 \item {\bf OBJECT CREATION \code{__call__}:}
+\item \textbf{OBJECT CREATION \code{__call__}:}
   If you write $R(x)$, then the \code{__call__} method of $x$ is called.  It
 should make an object of $R$ from $x$, if at all possible.
      \code{__call__}  is never called implicitly by binary operators.
     If $x$ already has $R$ as its  parent, then \code{__call__} must return
     a {\bf new copy} of $x$, unless $x$ is immutable (this is to agree with
+    a \textbf{new copy} of $x$, unless $x$ is immutable (this is to agree with
     standard Python conventions).
 For example, if $x \in \ZZ/n\ZZ$, then \code{ZZ.__call__(x)} is defined but
   \code{ZZ.__coerce__(x)} is not.
 \item {\bf ARITHMETIC \code{__add__}, \code{__mul__}, ...:}: When
+\item \textbf{ARITHMETIC \code{__add__}, \code{__mul__}, ...:}: When
   doing a binary operation, if the parents are not identical (in the
   sense of is\todo{incomplete?}), determine if precisely one {\tt
     \_coerce\_} map is defined; if so, apply it and do the arithmetic
   operation.  If both are defined, the parents are canonically
   isomorphic, so use the left one.  If neither are defined, raise a
   TypeError.  (Whether or not there is a coerce map between objects \todo{the objects?}
+  TypeError.  (Whether or not there is a coerce map between objects
   should be cached for efficiency.)
 \item {\bf PARENT OBJECT \code{__cmp__}}: \code{R == S} by definition
+\item \textbf{PARENT OBJECT \code{__cmp__}}: \code{R == S} by definition
   means that \code{_coerce_} is defined in both directions.  Roughly
   speaking this means that $R$ and $S$ are identical or canonically
   isomorphic, though there could be some exceptions to this.  Note
 …
   considered equal, since there are never canonical coercion maps in
   both directions when the categories are different.
 \item {\bf ELEMENT \code{__cmp__}}:
+\item \textbf{ELEMENT \code{__cmp__}}:
        If the parents aren't identical, test if
        precisely one \code{_coerce_} map is defined -- if so, return \code{__cmp__}
+       precisely one \code{_coerce_} map is defined --- if so, return \code{__cmp__}
        after applying the coerce.  If both coercions are defined, compute
        both \code{__cmp__}'s (in both $R$ and $S$).  Return the value if they give
        the same results.  Otherwise return $-1$ (the convention in Python
        is that \code{__cmp__} never raises an exception).   If no \code{_coerce_} is
        defined return $-1$ (i.e., not equal).
 \item {\bf IN \code{__contains__}}:
+\item \textbf{IN \code{__contains__}}:
        \code{x in R} should be true if and only if \code{R._coerce_(x)} would not
        raise a \code{TypeError}.
 \end{itemize}
 \section{Mutability}
 Parent structures (e.g., rings, fields, matrix spaces, etc.) should
 be immutable and globally unique whenever possible.   Immutability
 includes making it so properties like generator labels and default
+means, among other things, that properties like generator labels and default
 coercion precision cannot be changed.
 Global uniqueness while not wasting space storing objects that are not
 being referenced is best implemented using the standard Python weakref
+Global uniqueness while not wasting memory
+is best implemented using the standard Python weakref
 module, a factory function, and module scope
+variable.\todo{Rewrite. Difficult to parse. Make gentler}
+variable.
+\todo{Rewrite. Difficult to parse. Make gentler}
 \todo{Put a tutorial on this here}.
+\todo{Put a tutorial on this here}
 Certain objects, e.g., matrices, may start out mutable and become
 immutable later.  See the file \code{SAGE_ROOT/devel/sage/sage/structure/mutability.py}.
-\section{Importing Modules}
-Do not import at the top level of your module a third-party module that
-will take a long time to initialize (e.g., matplotlib).  It is important
-that \code{from sage.all import *} not take forever, since this is what
-dominates the \SAGE startup time.
-On a somewhat regular basis I do the following:
-\begin{verbatim}
-  echo "%prun import sage.all" | sage -ipython|more
-\end{verbatim}
-in order to profile what is taking a long time during
-the load.  In many cases I use the locate command to
-find files mentioned in the profile.
 \section{The {\tt \_\_hash\_\_} Special Method}
 Here's the definition of \code{__hash__}
 from the Python reference manual.
 …
 Python language, {\em which in \sage we simply cannot make} (!), and
 violating it has consequences.  Fortunately, the consequences are pretty
 clearly defined and reasonably easy to understand, so if you know
 about them they don't cause you trouble.  I think the following
+about them they don't cause you trouble.  The following
 example illustrates them pretty well:
 \begin{verbatim}
     sage: v = [Mod(2,7)]
 …
     KeyError: 9
 \end{verbatim}
 Here's another example,
+Here's another example:
 \begin{verbatim}
     sage: R = RealField(10000)
     sage: a = R(1) + R(10)^-100
 …
 Unfortunately, in \SAGE we simply can't require:
 \begin{verbatim}
           (*) "a == b ==> hash(a) == hash(b)"
+       (*)   "a == b ==> hash(a) == hash(b)"
 \end{verbatim}
 because serious mathematics is simply too complicated for this
 rule.    For example, the equalities
 …
 integers.
 The only way we could ``fix'' this problem for good would be to
 abandon using the "==" operator for "\SAGE equality", and implement
+abandon using the \code{==} operator for ``\SAGE equality'', and implement
 \SAGE equality as a new method attached to each object. Then we could
 follow Python rules for "==" and our rules for everything else, and
+follow Python rules for \code{==} and our rules for everything else, and
 all \SAGE code would become completely unreadable (and for that matter
 unwritable).  So we just have to live with it.
 So what is done in \sage is to {\em attempt} to satisfy (*) when it
+So what is done in \sage is to {\em attempt} to satisfy \verb+(*)+ when it
 is reasonably easy to do so, but use judgment and not go overboard.
 For example,
 \begin{verbatim}
     sage: hash(Mod(2,7))
 \end{verbatim}
 I.e., we get 2.  That's better than some weird random hash that also
+The output 2 is better than some random hash that also
 involves the moduli, but it's of course not right from the Python point
 of view, since \code{9 == Mod(2,7)}.
 The goal is to make a hash function that is fast but within reason
 respects any obvious, natural inclusions and coercions.
+respects any obvious natural inclusions and coercions.
+\section{Exceptions}
+\chapter{Creating a \SAGE Module or Package}
+\todo{What is a \SAGE Module and what is a \SAGE Package? Are they the
+  same? This chapter talks about only the latter? When
+  should/shouldn't one make a package out of a bunch of files?}
+\begin{enumerate}
+\item Design: Think about what your program will do and how that fits into the
+  structure of \SAGE.  In particular, much of \SAGE is implemented in
+  the object-oriented language Python, and there is a hierarchy of
+  classes that organize code and functionality.  It is {\em
+    surprisingly painful and difficult} to create some Python classes,
+  then realize that much functionality should be factored out into
+  base classes, and to do it\todo{Not sure what you are trying to
+    convey here}.  So try to get the structure right the first time.
+  For example, if you implement elements of a ring your class should
+  derive from
+  \class{sage.structure.element.RingElement}.\todo{Design is being
+    talked about without using the word. Consciously avoiding the
+    term?}
+\item Coding: Code it up, following the conventions in
+  Chapter~\ref{ch:conventions}.  Make the basic structure and
+  interface for what you're doing in Python.  You can use any of the
+  following languages to implement the hard parts: Python, C/C++,
+  Cython, Fortran 95, GAP, Singular, and GP/PARI.  You can also use the mwrank,
+  GSL, NTL, and PARI C/C++ libraries. (And if you are OK with your code
+  depending on optional \sage packages, you can use Octave, or
+  even Magma, Mathematica, or Maple.)  For readability and usability
+  it is best to use \sage when possible. On the other hand, for
+  efficiency reasons it is often crucial to use a compiled language or
+  an existing library or program---do not reinvent the wheel or write
+  code that is so slow as to be only a toy\todo{I split the
+    original line into two, as reusing existing code should be
+    emphasized more}.
+\item Documentation: Document {\em every} Python function that you've
+  defined,
+\item Testing: Liberally include examples in your source code. See
+  Chapter~\ref{ch:testing} for more on automated testing of examples.
+\item (Optional) Write a file that generates random input to test (and
+  break!) your code with the intention of making it robust (See
+  Chapter~\ref{ch:randomtesting}).
+\item (Optional) Send your package to the \code{sage-devel} Google group
+  for inclusion in \SAGE, or make a standalone
+  {\tt spkg} (see Chapter~\ref{ch:spkg}).
+\end{enumerate}
+\section{\SAGE Preparsing}
+\label{sec:Preparsing}
+The following is a list of files relevant to preparsing through which
+control flows when \sage is run:
+\begin{enumerate}
+\item \code{SAGE_ROOT/sage-sage},
+\item \code{SAGE_ROOT/sage-preparse},
+\item \code{SAGE_ROOT/devel/sage/sage/misc/preparser.py}
+  \todo{Talk about
+    \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py} file}
+\end{enumerate}
+The file preparser.py is contains the \SAGE preparser code and the
+following are some notes from the aforementioned file
+\begin{itemize}
+\item In \SAGE methods can also be called on integer and real literals
+  (note that in pure Python this would be a syntax error).
+\item Note that calling methods on int literals in pure Python is a
+  syntax error, but \SAGE allows this for \SAGE integers and reals,
+  because users frequently request it.
+\item Raw literals are not preparsed, which can be useful from an
+  efficiency point of view.  Just like Python ints are denoted by an
+  L, in \SAGE raw integer and floating literals are followed by an ``r''
+  (or ``R'') for raw, meaning not preparsed.
+\item Raw literals can be very useful in certain cases. For instance,
+  Python integers are typically much more efficient than \SAGE integers
+  when they are very small; large \SAGE integers are much more
+  efficient than Python integers, since they are implemented using the
+  GMP C library --- see section \ref{sec:Benchmarking}.
+\end{itemize}
+Consult preparser.py for details about \SAGE preparsing, examples involving raw literals, etc.
+When foo.sage is loaded in a sage session, a preparsed version of
+foo.sage is created titled foo.py. The beginning of foo.py states:
+\code{This file was *autogenerated* from the file foo.sage.}
+\section{Randomized Testing}
+\label{ch:randomtesting}
+In addition to all the examples, which serve as both demonstrations
+and tests of your code, you should also create a test suite.  Think of
+this as a program that will run for a while and ``tries'' to crash
+your code using randomly generated input.  Your test code should
+define a class \class{Test} with a {\tt random()} method that runs
+random tests.  These are all assembled together later, and each test
+is run for a certain amount of time on a regular basis.
+\section{Creating a New \sage spkg Package}
+\label{ch:spkg}
+Here's how to make your own \sage package.
+\begin{itemize}
+\item Note that a \sage package is simply a .tar.bz2 file (or a tar
+  file), but named with .spkg to discourage confusion.  In particular,
+  you can type
+Please avoid code like this:
 %skip
 \begin{verbatim}
+            tar jxvf my_package-version.spkg
+try:
+    some code
+except:               # bad
+    more code
 \end{verbatim}
+to extract one and see what is inside.
+Instead catch specific exceptions. For example,
+\begin{verbatim}
+try:
+    return self.__coordinate_ring
+except (AttributeError, other exceptions), msg:           # Good
+    more code to compute something
+\end{verbatim}
+Note that the syntax in \code{except} is to list all the exceptions
+that are caught as a tuple, followed by an error message.
+\item
+To create your own package:
+\begin{itemize}
+\item[(a)]
+Make a directory, e.g., \code{mypackage-0.1}.
+The name of the directory should be a lower-case string with no
+dashes followed by a dash followed by a version number.
+If you don't have any exceptions explicitly listed (as a tuple), your
+code will catch absolutely anything, including \code{ctrl-C} and alarms,
+and this will lead to confusion.
-\item[(b)]
-Put your files in that directory.
+\item[(c)]
+Create an executable shell script \code{mypackage-0.1/spkg-install}.
+\section{Importing}
+\item[(d)]
+The script \code{spkg-install} is run during installation
+of the \sage package.   In this script you may
+make the following assumptions:
+\begin{itemize}
+\item
+The PATH has \code{sage} and the \sage install's \code{python}
+location at the front.  Thus the command
+We mention two issues with importing: circular imports and importing
+large third-party modules.
+First, you must avoid circular imports.  For example, suppose that the
+file \code{SAGE_ROOT/devel/sage/sage/algebras/steenrod_algebra.py}
+started with a line
 %skip
 \begin{verbatim}
+                     python setup.py install
+from sage.sage.algebras.steenrod_algebra_bases import *
 \end{verbatim}
+will run the correct version of Python with
+everything setup correctly.  Also, e.g.,
+running \code{gap} or \code{Singular} will run the
+correct versions.
+\item
+The environment variable \code{SAGE_ROOT} points to
+the root directory of the \sage install.
+\item
+The environment variable \code{SAGE_LOCAL} points to
+the \code{SAGE_ROOT/local} directory of the \sage install.
+\item
+The environment variables
+\code{LD_LIBRARY_PATH} and \code{DYLD_LIBRARY_PATH}
+both have \code{SAGE_ROOT/local/lib} at the front.
+\end{itemize}
+\item[(e)]
+The \code{spkg-install} script should copy your files to the appropriate
+place after doing any build that is necessary.  That's it!
+\item[(f)] (Optional) Send a copy to the \code{sage-devel} Google group
+so it can be posted to the \SAGE web site, so anybody can automatically
+install it by typing \code{sage -i my_package-version.spkg}.
+\end{itemize}
+\end{itemize}
+For example, the location of GAP is \code{SAGE_ROOT/local/lib/gap-4.4.6/}.
+Thus to install gap code you have to copy it there.  You should check
+that the directory with the given version your package is created for
+exists in your \code{spkg-install} script, e.g., with the following code:
+and that the
+file \code{SAGE_ROOT/devel/sage/sage/algebras/steenrod_algebra_bases.py}
+started with a line
 %skip
 \begin{verbatim}
+if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.6 ]; then
+    echo "This package requires that GAP 4.4.6 is installed."
+    exit 1
+fi
+from sage.sage.algebras.steenrod_algebra import SteenrodAlgebra
+\end{verbatim}
+This sets up a loop: loading one of these files requires the other,
+which then requires the first, etc.
+With this set-up, running \sage will produce an error:
+%skip
+\begin{verbatim}
+Exception exceptions.ImportError: 'cannot import name SteenrodAlgebra' in 'sage.rings.polynomial.polynomial_element.Polynomial_generic_dense.__normalize' ignored
+---------------------------------------------------------------------------
+ImportError                               Traceback (most recent call last)
+...
+ImportError: cannot import name SteenrodAlgebra
+\end{verbatim}
+Instead, you might replace the \code{import *} line at the top of the
+file by more specific imports where they are needed in the code
+-- e.g., the \code{basis} method for the class \code{SteenrodAlgebra}
+might look like this (omitting the documentation string):
+%skip
+\begin{verbatim}
+    def basis(self, n):
+        from steenrod_algebra_bases import steenrod_algebra_basis
+        return steenrod_algebra_basis(n, basis=self._basis_name, p=self.prime)
 \end{verbatim}
+\emph{Caveat}: Do {\em not} just
+copy to \code{SAGE_ROOT/local/lib/gap*/} since that will copy
+your package to the lib directory of the {\em old} version of GAP if
 GAP is upgraded.
+Second, do not import at the top level of your module a third-party
+module that will take a long time to initialize (e.g., matplotlib).
+As above, you might instead import specific components of the module
+when they are needed, rather than at the top level of your file.
+External Magma code goes in
+\verb+SAGE_ROOT/data/extcode/magma/user+, so if you want to redistribute
+Magma code with \sage as a package that Magma-enabled users can use,
+that's where you'd put it.  You'd also want to have relevant Python code
+to make the Magma code easily usable.
+It is important to try to make \code{from sage.all import *} as fast
+as possible, since this is what dominates the \SAGE startup time, and
+controlling the top-level imports helps to do this.
-{\bf Example}:
-These instructions provide directions for creating a particular
-\sage package. \todo{Something about this example bugs me and I'm not sure what it is.}
+\begin{itemize}
+\item
+Download \code{guava2.4.tar.gz} (the tarball for GUAVA, the
+GAP error-correcting codes package) from
+\code{http://www.gap-system.org/Packages/guava.html}.
+\section{Using Optional Packages}
+\item
+Suppose you have a directory \code{sage-package-guava}.
+Within that, create a directory \code{guava-2.4}.
+Extract the tarball into that directory
+(so there will be a \code{guava2.4} subdirectory of
+\code{sage-package-guava}).
+If a function requires an optional package, that function should fail gracefully ---
+perhaps using a \code{try} and \code{except} block --- when the optional
+package is not available, and should give a hint about how to install
+it.  For example, typing \code{sage -optional} gives a list of all
+optional packages, so it might suggest to the user that they type that.
-\item
-Put the following script called \code{spkg-install} in
-\code{sage-package-guava}:
-%skip
-\begin{verbatim}
-#!/bin/sh
+if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.6 ]; then
+    echo "This package requires that GAP 4.4.6 is installed."
+    exit 1
+fi
+\chapter{Coding in Other Languages}
+cp -pr guava2.4 $SAGE_ROOT/local/lib/gap-4.4.6/pkg/
+When writing code for \sage, use Python for the basic structure and
+interface.  For speed, efficiency, or convenience, you can implement
+parts of the code using any of the following languages:
+Cython, C/C++, Fortran 95, GAP, Singular, and GP/PARI.  You can also
+use the mwrank, GSL, NTL, and PARI C/C++ libraries. (And if you are
+okay with your code depending on optional \sage packages, you can use
+Octave, or even Magma, Mathematica, or Maple.)
+cd $SAGE_ROOT/local/lib/gap-4.4.6/pkg/guava2.4
+./configure ../..
+make
 \end{verbatim}
+The first section of this chapter discusses Cython (which is a
+compiled language, based on Python); many components of \sage are
+written in Cython.  Later sections discuss the interfaces between
+\sage and PARI, GAP, and Singular.
-Make it executable with \code{chmod +x spkg-install}.
-\item
-In the directory \code{sagefiles},
-issue the command
+%skip
+\begin{verbatim}
+<path>sage -pkg guava-2.4
+\end{verbatim}
+where \verb+<path>+ means the absolute path to the sage
+script. This will do some checks, run tar/bz2, and create the spkg file,
+\code{guava-2.4.spkg}, in the
+directory \code{sagefiles},
+\end{itemize}
+\section{Cython}
+To test this, within the \code{sagefiles} directory
+(or whatever directory contains \code{guava-2.4.spkg}),
+type
+Cython is a compiled version of Python; it is based on Pyrex
+(\url{http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/}).  To
+a large degree, Cython has changed based on what \sage's developers
+needed: Cython has been developed in concert with \sage.
+%skip
+\begin{verbatim}
+<path>sage -i guava-2.4.spkg
+\end{verbatim}
+It should install and compile the guava program in sage.
+To test its functionality, one could do the following:
+As such, it is a young, but developing, language, with young, but
+developing, documentation.  See its web page,
+\url{http://www.cython.org/}, for the most up-to-date information.
+\begin{verbatim}
+sage: gap(`LoadPackage("guava")')          # requires optional package
+true
+sage: gap(`HammingCode(3,3)')              # requires optional package
+a linear [13,10,3]1 Hamming (3,3) code over GF(3)
+\end{verbatim}
+To  build again if you've already installed the package, use
+%skip
+\begin{verbatim}
+<path>sage -f guava-2.4.spkg
+\end{verbatim}
+\chapter{Documentation}
+\section{Source Code Documentation}
+Use \Latex formatting in the documentation string.  If you wish to use
+backslashes in the documentation string, use double backslashes or place an {\tt r} right before the first triple opening quote. For example,
+%skip
+\begin{verbatim}
+def cos(x):
+    """
+    Returns $\\cos(x)$.
+    """
+def sin(x):
+    r"""
+    Returns $\sin(x)$.
+    """
+\end{verbatim}
+The non-standard macros that are used when typesetting the
+documentation are defined in
+\code{SAGE_ROOT/doc/commontex/macros.tex}.  If you need to add more macros,
+add them there and post them on the \code{sage-devel} Google group so that
+they can be included in the next version of \sage.
+The first line two or three lines of the docstring should briefly summarize
+what the function does, e.g., ``Returns X'' or ``Computes X'',
+or ``Creates a new X''.  Next describe the input and output of
+the function using the following notation:
+%skip
+\begin{verbatim}
+INPUT:
+    var1 -- <type> (default: ...) description
+    var2 -- <type> (default: ...) description
+    etc.
+OUTPUT:
+    <type> -- description of return value 1
+    <type> -- description of return value 2
+    etc.
+\end{verbatim}
+This is a placeholder of a "Perfect example of the above".\todo{
+I=ifti propose everywhere in this document when there is a template, it is followed by a path/filename where in one can find a "perfect" example.}
+For consistency you should use the above format, which is used
+throughout the system.  Do not just describe the input and output
+in long sentences.    For methods of a class, whether or not
+to describe self is up to you; this can be useful if self must
+have some special properties.  Also, you do not have to
+specify the types of the input and output arguments
+precisely; e.g., use ``integer'' for an int or long or
+\sage \code{Integer}.
+Every function should have at least one, and preferably many, examples
+that illustrate its usage. These examples must exactly match the
+output of an actual running copy of Python.  To test whether this is
+the case, run the program \code{sage -t} with your Python or Cython
+file as unique argument (see Chapter~\ref{ch:testing} for more details).
+For more details, see Section~\ref{sec:conv-docs}.\todo{redundancy \S 2.4 and \S4.1}
+\section{Extra Documentation}
+You might also write additional documentation in \Latex, which is not to
+be part of any source code file.  The examples in this documentation
+should be contained in verbatim environments.  The examples should be
+tested using \code{sage-testtex}, which you run on the \Latex file.  Put
+\%skip on the line preceding the environment to skip testing of an
+example.
+Note that \code{sage-testtex} is completely different than \code{sage-doctest}.  It extracts the documentation and feeds it to a
+running instance of \SAGE.
+\section{Automated Testing}\label{ch:testing}
+This section describes automated testing of test files of the
+following types: \code{.py .pyx .sage .tex}.  Use \code{sage -t
+  <file>} to test that the examples in \code{<file>} behave exactly as
+claimed.  See Section~\ref{sec:conv-docs} for a discussion of how to
+include examples in documentation strings and what conventions to
+follow --- Section ~\ref{sec:Further-conventions}.
+\subsection{Testing .py, .pyx and .sage Files}\label{sec:test:python}
+Run \code{sage -t <filename.py>} to test that all code examples in
+\code{filename.py}.  Similar remarks apply to \code{.sage} and
+\code{.pyx} files.
+\begin{verbatim}
+  sage -t [--verbose] [--optional]  [files and directories ... ]
+\end{verbatim}
+When you run \code{sage -t <filename.py>}, \sage makes a copy of
+\code{<filename.py>} with all the \code{sage} prompts replaced by
+\code{{>}{>}>}, then uses the standard Python doctest framework to test
+the documentation.  More precisely,
+the Python script \code{local/bin/sage-doctest} implements
+documentation testing.  It does the following when asked
+to test a file \code{foo.py} or \code{foo.sage}.
+\begin{enumerate}
+\item Creates the directory \code{.doctest} if it doesn't
+exist and the file \code{.doctest/foo.py}.
+\item The file \code{.doctest_foo.py} contains functions for
+each docstring in \code{foo.py}, but with all \sage preparsing
+applied and with \code{from sage.all import *} at the top.
+The function documentation is thus standard Python with
+\code{{>}{>}>} prompts.
+\item \code{sage-doctest} then runs \sage's Python interpreter
+on \code{.doctest_foo.py}.
+\end{enumerate}
+\note{{\em Note that line numbers in the errors
+you might see apply to the file \code{.doctest_foo.py}, not
+to the original file \code{foo.py}!}  If you get errors,
+feel free to inspect \code{.doctest_foo.py}; it is never
+automatically deleted by \sage.}
+Your file passes these tests if the code in it will run when entered
+at the \code{sage:} prompt with no special imports.  Thus users are
+guaranteed to be able to exactly copy code out of the examples you
+write for the documentation and have them work.
+%skip
+\begin{verbatim}
+William:
+I changed the doctest behavior recently so that now you can doctest
+files anywhere (not necessary in \SAGE), and type e.g.,
+    sage: from foo import *
+    sage: ...
+in the doctest blocks and it will work, even if foo.py isn't part of the
+\SAGE library.  It's much better now.   Feel free to change the reference
+manual to reflect this.
+\end{verbatim}
+\subsection{Testing \Latex Documentation}
+Run \code{sage -t <filename.tex>} to test the examples in
+verbatim environments in \Latex documentation.   Put
+\code{\%skip}
+on the line right before a verbatim environment to have that
+example skipped when testing the file.
+\sage creates a file \code{.doctest_filename.py} and tests
+it just as for \code{.py}, \code{.pyx} and \code{.sage}
+files.   In order to skip testing of a block of code
+in a verbatim environment, put the \Latex comment
+\code{\%skip} on the previous line.
+One difference is that in \Latex files one
+often inserts explanatory texts between different
+verbatim environments.    To link together verbatim
+environments use the \code{\%link} comment.  For
+example\todo{I have changed verbatim to verbatim0 here
+since I don't know how to include end{verbatim} in a
+verbatim environment.}
+%skip
+\begin{verbatim}
+    \begin{verbatim}
+    sage: a = 1
+    \end{verbatim_}%link
+    Next we add 1 to \code{a}.
+    %link
+    \begin{verbatim}
+    sage: 1 + a
+    \end{verbatim_}
+\end{verbatim}
+See \code{SAGE_ROOT/doc/tut/tut.tex} for many examples
+of how to include automated testing in \Latex documentation
+for \sage.
+\section{Modifying the \SAGE Manuals}
+We use the tutorial as an example.
+You just modify the file \code{SAGE_ROOT/devel/doc/tut/tut.tex} To
+build a test copy of the tutorial with your changes, type
+\code{./build_dvi} in the \code{devel/doc/tut} directory.  You'll get
+a file \code{tut.dvi}.
+You can also build HTML versions of everything by typing \code{make
+  html} in the \code{devel/doc} directory.  Build pdfs with \code{make
+  pdf}. \todo{You could also do \code{./build_pdf} to create a pdf directly from the tex file.}
+It is a good idea to type \code{sage -t tut.tex} to make sure any
+examples you added to the tutorial work as claimed.
+The \code{SAGE_ROOT/devel/doc} directory is a Mercurial repository
+(see Chapter~\ref{ch:mercurial}), so you can see exactly what changes
+you've made to the documentation, make patches of what you've done
+available, etc.  You can do this from within \sage by typing
+\code{hg_doc.[tab]}.
+Imagine you have edited this Programming Manual found in \code{devel/doc/prog}.
+Before you started you did a pull to be sure your docs are up to date:
+\todo{Give some examples of usage}
+%skip
+\begin{verbatim}
+sage: hg_doc.pull()
+cd "/home/jaap/sage/devel/doc" && hg status
+cd "/home/jaap/sage/devel/doc" && hg status
+cd "/home/jaap/sage/devel/doc" && hg pull -u http://sagemath.org/sage//hg//doc-main
+pulling from http://sagemath.org/sage//hg//doc-main
+searching for changes
+adding changesets
+adding manifests
+adding file changes
+added 3 changesets with 3 changes to 1 files
+files updated, 0 files merged, 0 files removed, 0 files unresolved
+If it says use 'hg merge' above, then you should
+type hg_sage.merge(), where hg_sage is the name
+of the repository you are using.  This might not
+work with the notebook yet.
+\end{verbatim}
+After you made many changes, you want to commit them:
+%skip
+\begin{verbatim}
+sage: hg_doc.commit()
+cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg diff  | less
+cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg commit
+\end{verbatim}
+You saw the differences piped through \code{diff}, you were asked
+to comment: type 'i' or 'a', your comment, hit 'Escape' and type ':wq".
+Now you can make a patch 'patchname'. e.g.:
+%skip
+\begin{verbatim}
+sage: hg_doc.send('prog20070118.hg')
+Writing to /home/jaap/sage/prog20070118.hg
+cd "/home/jaap/sage/devel/doc" && hg bundle  tmphg http://sagemath.org/sage//hg//doc-main
+searching for changes
+Successfully created hg patch bundle /home/jaap/sage/prog20070118.hg
+\end{verbatim}
+Send the file 'patchname' to the \code{sage-devel} Google group.
+\chapter{Writing Optimized Compiled Code With \SAGE}
+This chapter is about how to write fast compiled optimized code for
+\sage.  It was written mainly by William Stein with contributions
+from David Harvey, Martin Albrecht, David Joyner, Alex Clemesha.
+Some of the text is closely copied or translated (to mathematical
+language) from Greg Ewing's original Pyrex manual.
+This chapter assumes that you have a C compiler suite (e.g., GCC)
+installed on your computer, since you should not write compiled
+extension code on a computer without a C compiler.  If you are using
+OS X, download and install the latest version of XCode.\footnote{The
+  download is about 1GB, and you must sign up for the Apple Developer
+  Network. \url{http://developer.apple.com/tools/xcode/}}
+For Linux, use
+your package manager to install GCC. If you're using Microsoft
+Windows, then you should be using \sage via colinux, which includes
+GCC by default.
+%[[General discussion of when to use compiled code versus interpreted
+%code.]]
+Interpreted code: way easier to debug,
+trace, document, structure (multiple inheritence), etc.;
+good when few for loops; lots of structure and calls
+to compiled code.
+Compiled code: two reasons to use---make fast code and
+make optimal use of libraries written in a compiled language;
+hard to write and debug; takes longer to
+write.  The methods explained in this chapter can produce
+code where speed is only limited by your choice of algorithm.
+{\bf TODO:}
+\begin{enumerate}
+\item Module name and file name must agree!
+%[(Check this at build time?)]
+\end{enumerate}
+\section{Introduction}
+This chapter describes how to write \SAGE programs using Cython, which
+is a language for writing Python extension modules that is a very
+slightly variant of Pyrex (we will often refer to it as ``Pyrex''
+still).
+The primary goal of this chapter is to provide excellent documentation
+about how to use Cython to write very fast code in the context of
+\SAGE.  (Currently it is a mess.)  This chapter is self contained in
+that the reader should not have to look at any of the documentation
+distributed with Pyrex.  The reader is urged to at some point read the
+Python/C API Reference manual.
+There is also much to be learned from the Pyrex
+mailing list:
+\url{http://lists.copyleft.no/mailman/listinfo/pyrex}.
+{\bf Acknowledgement:} The existence of Greg Ewing's Pyrex is
+one of the primary reasons Stein chose Python as the implementation
+language for \SAGE.  It is a brilliantly useful tool.
+%\cite{pyrex:list}.
+%A list of quality open source projects that
+%use Pyrex: soya3d, ... [[add more here]]
+\subsection{Interpreted and Compiled Code}
+Every serious computer algebra system, e.g., MAGMA, PARI, Mathematica,
+Maple, GAP, Singular, etc., is implemented as a combination of
+compiled and interpreted code.  For Mathematica, Singular, MAGMA, and
+PARI, most of the implementation is in compiled C (or C++) code; some
+of these systems tend to be very optimized (subject to the constraints
+of the algorithms they implement). In contrast, Maple and GAP have a
+relatively small compiled ``kernel'' that defines the underlying
+programming language; most of the system is then implemented in this
+language.  If you do benchmarks you'll discover that Mathematica is
+{\em much} faster than Maple at some basic operations.  Likewise,
+benchmarks reveal that MAGMA is often faster than GAP.
+%[[give
+%examples that illustrate each of these points.  E.g., basic
+%arithmetic where maple is pitiful, computation of Bernoulli
+%numbers where PARI crushes all others.]]
+This fusion of interpreted and compiled code is extremely natural for
+mathematics software; some algorithms are much better implemented in
+an interpreter because all time-critical steps involve low level
+arithmetic --- other algorithms, e.g., matrix multiplication, must be
+implemented in a compiled language in order to perform optimally.
+Also, existing compiled libraries are sometimes of very high quality,
+and a compiled language is needed to create the best possible
+interface to such libraries.  It's crucial that both approaches to
+programming be fully supported.  When deciding how to implement \SAGE,
+I searched for months for an environment that could support both
+approaches.  Python and Cython provide exactly this in a way that I
+believe is much easier conceptually than the implementation models of
+any of the other systems mentioned above.
+\subsection{Why Cython is the Best Available Option for \SAGE}
+The software listed in this section is used either
+to speed up Python programs or to make code written in
+C++, C, FORTRAN, etc., available in Python.   This section briefly
+explains what each tool does and why it is {\em not} included
+in \SAGE or used in the implementation of the \SAGE library.
+\subsubsection{SWIG}
+SWIG (Standard Wrapper and Interface Generator)
+takes C++ (and C, etc.) libraries, and generates
+wrapper code so that the code can be used from
+within Python (and in several other interpreters).  One
+writes interface files that describe how to make Python
+objects usable from your C++ code, and conversely what
+transformations C++ objects should undergo to be usable from Python.
+In 2004, Stein considered using SWIG for \SAGE extensively, and implemented
+several basic types using this model.  The idea was to
+write code in C++ for \SAGE that needed to be fast, then wrap it in
+SWIG.  This ground to a halt, because the result was not
+  {\em sufficiently} fast.  First, there is overhead when writing code
+in C++ in the first place.  Second, SWIG generates several layers of
+code between Python and the code that does the actual work; for
+implementing any of \SAGE's numerous basic arithmetic types,
+e.g., integers, the overhead is completely unacceptable, at
+least if one wants code that is as fast as Magma and PARI.  For
+example, in my initial implementation of integer arithmetic using
+SWIG and GMP, there were the following {\em six} levels between the
+\sage interpreter and the GMP C library:
+\begin{center}
+\begin{tabular}{|l|}\hline
+Python code to provide a clean interface\\\hline
+SWIG Autogenerated Python code \\\hline
+SWIG Autogenerated C++ extension code \\\hline
+Handcode C++ Integer class\\\hline
+GMP's C++ Interface\\\hline
+GMP C Library\\\hline
+\end{tabular}
+\end{center}
+In this model, just multiplying two integers is extremely complicated,
+and slow.  Moreover, writing the SWIG interface files was
+painful, since general Python types are difficult to work with from
+C++ code.  In contrast, with Cython the situation is as follows
+(see {\tt integer.pyx}):
+\begin{center}
+\begin{tabular}{|l|}\hline
+Cython generated C code that provides everything needed\\\hline
+GMP C Library\\\hline
+\end{tabular}
+\end{center}
+Another important problem with using SWIG is that
+developers have to know C++ well and have the mental dexterity
+to easily switch back and forth between C++ and Python.
+Stein does not have this ability.
+\subsubsection{Boost}
+Like SWIG, Boost takes C++ code and generates C/C++ extension code and Python
+wrappers around that extension code, which one can then use from the
+Python interpreter.  \SAGE does not use Boost because it is large and
+difficult to redistribute and it is meant much more for wrapping
+existing C++ libraries.  Boost also uses a very complicated and
+specialized build system.
+\subsubsection{Psycho}
+Psycho is a Python library that tries to guess the types of variables
+in a running Python program, then generate compiled extension code
+{\em on the fly} to optimize the running program.  \SAGE does not use
+Psycho, because it only runs on Intel x86 machines,
+%[[what about
+%OS X macs, windows?]]
+it has large and
+unpredictable memory requirements, and so much behind-the-scenes
+guessing at this level for a serious research mathematics tool is
+inappropriate.
+%[[Nonetheless: quick tutorial and examples]]
+Moreover, in my own experience with \sage, the
+speedups from Psycho were not significant, except on code that had an
+obvious C equivalent.  Such code can be easily rewritten in Cython,
+which always results in something that is faster, more robust, and
+more reliable than what Pyscho produces, and which can be debugged
+using gdb.
+Though Psycho is not included with or used by \SAGE,
+it is possible to try out Psycho by
+installing it and putting the appropriate line at the top of a file.
+%[[put a mathy example in with timings]]
+\subsubsection{Ctypes}
+Ctypes is included standard in Python 2.5, so it is clearly useful to
+many Python users.  It allows one to call library
+functions defined in shared object libraries directory from
+interpreted Python code.  For the core of \SAGE
+we have not found a use for this yet.
+First, when implementing a basic type that needs to
+be very fast, e.g., GMP integers, the entire implementation absolutely
+must be compiled -- writing it as a combination of Python and C
+library calls will be way too slow.  Second, it is often very painful
+to convert Python data structures to C data structures and conversely,
+and there are many opportunities for memory leaks and core dumps.
+%[[example.  like the ``install'' command in PARI.]]
+\section{Examples}
+Python is an interpreted language and has no declared data types for
+variables.  These features make it easy to write and debug, but Python
+code can sometimes be slow.  Cython code can look a lot like Python,
+but it gets translated into C code (often very efficient C code) and
+then compiled.  Thus it offers a language which is familiar to Python
+developers, but with the potential for much greater speed.
 There are several ways to create and build Cython code in \SAGE.
 \begin{enumerate}
 …
 for you to use in the notebook.  Also, the output cell has
 a  link to the C program that was compiled to create
 the .so file.
-\item Below we will denote notebook sessions as follows:
-\begin{verbatim}
-{{{
-  INPUT TEXT
-///
-  OUTPUT TEXT
-}}}
-...
-\end{verbatim}
-The triple braces delimit cells.  You should {\em not}
-actually type in the curly braces or triple forward
-slashes.
 \end{enumerate}
 \item  Create an .spyx file and attach or load it
 from the command line.
 This is similar to creating a {\tt \%cython} cell
 in the notebook but works completely from the command
 line.
+line (and not from the notebook).
+\item Create a .pyx file in the \SAGE library, add it to {\tt
+    SAGE_ROOT/devel/sage/setup.py} (note -- you have to follow the
+  template of how something else in there is done -- we simply haven't
+  had the time to properly document this process yet!), and type {\tt
+    sage -br} to build it.  This is how much of the core arithmetic in
+  the \SAGE library is implemented.
+\item Create a .pyx file in the \SAGE library and add a listing for it
+to the variable \code{ext_modules} in the file
+\code{SAGE_ROOT/devel/sage/setup.py}.  For example, the file
+\code{SAGE_ROOT/devel/sage/sage/graphs/chrompoly.pyx} has lines
+%skip
+\begin{verbatim}
+    Extension('sage.graphs.chrompoly',
+              ['sage/graphs/chrompoly.pyx']
+              ), \
+\end{verbatim}
+in \code{setup.py}.  Then type \code{sage -ba} to build \sage with the
+new code.
 \end{enumerate}
 \subsection{Attaching or Loading {\tt .spyx} Files}
 The easiest way to try out Cython without having to
 learn anything about distutils, etc., is to create
 a file with the extension {\tt spyx}, which stands
 for ``SAGE Pyrex''.   Try it now!
+for ``SAGE Pyrex'':
-\subsection{Using the Command Line and a File}
 \begin{enumerate}
 \item Create a file {\tt hello.spyx}.
+\item Create a file \code{power2.spyx}.
 \item Put the following in it:
 \begin{verbatim}
+def hello(name):
+    """
+    Print hello with the given name.
+    """
+    print("Hello %s"%name)
+def is2pow(n):
+    while n != 0 and n%2 == 0:
+        n = n >> 1
+    return n == 1
 \end{verbatim}
 \item Start the \SAGE command-line interpreter and load
 the spyx file (this will fail if you do not have a C compiler
 installed).
 %skip
 \begin{verbatim}
 sage: load "hello.spyx"
 Compiling hello.spyx...
 sage: hello('World')
+Hello World
+sage: load "power2.spyx"
+Compiling power2.spyx...
+sage: is2pow(12)
+False
 \end{verbatim}
 \end{enumerate}
 Note that you can change {\tt hello.spyx}, then load
+Note that you can change \code{power2.spyx}, then load
 it again and it will be recompiled on the fly.
 You can also attach {\tt hello.spyx} so it is reloaded
 whenever you make changes.
+You can also attach \code{power2.spyx} so it is reloaded
+whenever you make changes:
 %skip
 \begin{verbatim}
 sage: attach "hello.spyx"
+sage: attach "power2.spyx"
 \end{verbatim}
+\subsection{An Optimization Example}
+Imagine you are writing a program that frequently computes {\tt a*b + c}
+for $a,b,c$ \SAGE integers.  For example, you want to
+do the following computation (timings on a Macbook Pro 2Ghz
+under OS X):
+Cython is used for its speed.  Here is a timed test on a 2.4 GHz iMac:
 %skip
 \begin{verbatim}
 sage: a = 939393; b = 19393; c = 39390283
+sage: time for _ in range(10000): a = a*b + c
 CPU times: user 0.33 s, sys: 0.21 s, total: 0.53 s
+sage: time [n for n in range(10^5) if is2pow(n)]
+[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
+CPU time: 0.06 s,  Wall time: 0.06 s
 \end{verbatim}
+There is no ``multiply then add'' operation built into \SAGE, and
+you're very frustrated because you really want this to be as fast as
+possible.  There is possibly significant overhead in the computation
+of {\tt a*b + c} in Python.
+First create a new compiled ``multiply and add'' command in \SAGE.
+%[[todo: change example below to use {\tt\_\_new\_\_}.]]
+\begin{verbatim}
+cimport sage.rings.integer
+def muladd(sage.rings.integer.Integer a,
+           sage.rings.integer.Integer b,
+           sage.rings.integer.Integer c):
+    """
+    Compute a*b + c
+    """
+    cdef sage.rings.integer.Integer d
+    d = sage.rings.integer.Integer()
+    mpz_mul(d.value, a.value, b.value)
+    mpz_add(d.value, d.value, c.value)
+    return d
+\end{verbatim}
+Do this either by putting the above code in a spyx
+file or pasting it into the notebook in a cell that
+starts with {\tt\%cython}.
+Now we can do the above computation more quickly:
+Now, the code in the file \code{power2.spyx} is valid Python, and if
+we copy this to a file \code{powerslow.py} and load that, we get the
+following:
 %skip
 \begin{verbatim}
+sage: a = 939393; b = 19393; c = 39390283
+sage: time for _ in range(10000): a = muladd(a,b,c)
+CPU times: user 0.11 s, sys: 0.05 s, total: 0.15 s
+sage: load "powerslow.py"
+sage: time [n for n in range(10^5) if is2pow(n)]
+[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
+CPU time: 1.80 s,  Wall time: 1.84 s
 \end{verbatim}
+By the way, we could gain even a little more speed -- cutting it down
+to 0.03 s -- with the Cython version with a type declaration, by
+changing \code{def is2pow(n):} to \code{def is2pow(unsigned int n):}.
-To squeeze out even more performance we can put the
-entire loop in a single function.
-\begin{verbatim}
-cimport sage.rings.integer
-def muladdloop(sage.rings.integer.Integer a,
-               sage.rings.integer.Integer b,
-               sage.rings.integer.Integer c,
-               int n):
-    cdef int i
-    cdef mpz_t t
-    cdef sage.rings.integer.Integer aa
-    aa = sage.rings.integer.Integer(a)
-    mpz_init(t)
-    for i from 0 <= i < n:
-        mpz_mul(t, aa.value, b.value)
-        mpz_add(aa.value, t, c.value)
+    mpz_clear(t)
+    return aa
+\end{verbatim}
+\section{Other Languages}
+We then have
+%skip
+\begin{verbatim}
+sage: a = 939393; b = 19393; c = 39390283
+sage: time z=muladdloop(a,b,c,10000)
+CPU times: user 0.11 s, sys: 0.00 s, total: 0.11 s
+\end{verbatim}
+This is a little faster. %[[no system overhead -- what's going on?]]
+Since \sage is based on Python, it interfaces with C and C++, as well
+as other languages.  See the Python documentation at
+\url{http://www.python.org/doc/} for more details; in particular,
+``Extending and Embedding the Python Interpreter'', available at
+\url{http://docs.python.org/ext/ext.html}, describes how to write C or
+C++ modules for use in Python.
-\subsection{A simple loop example: Sum of squares}
-The following \SAGE worksheet illustrates the above
-sums of squares examples in the \SAGE Notebook.
-Note, we use the notation:
-%skip
-\begin{verbatim}
-{{{
-  INPUT TEXT
-///
-  OUTPUT TEXT
-}}}
-\end{verbatim}
-to denote cells in the notebook.
-%[[todo put a check to avoid overflow below.  maybe switch
-%to using long long, or also use that...]]
-\begin{verbatim}
-{{{
-two = int(2)
-def sumsquarespy(n):
-    return sum(i**two for i in xrange(1,n+1))
-}}}
-{{{
-time v=[sumsquarespy(100) for _ in xrange(10000)]
-///
-#> CPU time: 0.49 s,  Wall time: 0.49 s
-}}}
-{{{
-%cython
-# Next in cython.
-def sumsquares(int n):
-    cdef int i, j
-    j = 0
-    for i from 1 <= i <= n:
-        j = j + i*i
-    return j
-}}}
-{{{
-time v=[sumsquares(100) for _ in xrange(10000)]
-///
-CPU time: 0.06 s,  Wall time: 0.06 s
-}}}
-\end{verbatim}
-\subsection{Example: 2 Power}
-The following \SAGE notebook session illustrates how to implement
-in Cython (entirely using the notebook) a function to quickly tell
-if a C integer is a power of $2$.  The resulting function is over 200
-times faster in Cython than in Python, though the actual code
-is {\em identical}.
-\begin{verbatim}
-{{{
-%cython
-def is2pow(unsigned int n):
-    while n != 0 and n%2 == 0:
-        n = n >> 1
-    return n == 1
-}}}
-\end{verbatim}
-\begin{verbatim}
-{{{
-time [n for n in range(10^5) if is2pow(n)]
-///
-[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
-CPU time: 0.05 s,  Wall time: 0.06 s
-}}}
-\end{verbatim}
-\begin{verbatim}
-{{{
-# The same program but in Python:
-def is2pow(n):
-    while n != 0 and n%2 == 0:
-        n = n >> 1
-    return n == 1
-}}}
-\end{verbatim}
-\begin{verbatim}
-{{{
-time [n for n in range(10^5) if is2pow(n)]
-///
-[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
-CPU time: 13.04 s,  Wall time: 15.08 s
-}}}
-\end{verbatim}
-\begin{verbatim}
-{{{
-.04/0.05
-///
-.79999999999995
-}}}
-\end{verbatim}
-%\subsection{Adding to \SAGE}
-%\subsection{Your Own {\tt setup.py}}
-\section{Cython Manual}
-\subsection{Memory Usage}
-Here is the memory usage of an integer mod int object on a 32-bit system:
-\begin{verbatim}
-an int (the underlying value)
-pointer to the parent
-pointer to the modulus
-pointer to the function call table
-PyObject_HEAD  (pointer to type object and reference count)
-\end{verbatim}
-So a single integer mod takes 24 bytes.
-{\em IDEA: For a small field, one could store every element, then just have
-lots of copies of pointers to the same thing, which would save a ton
-of memory. }
-\subsection{Forward-declaring extension types}
-Extension types can be forward declared, which is necessary, e.g.,
-if you have two extension types that need to refer to each other.
-For example,
-\begin{verbatim}
-cdef class MyRing # forward declaration
-cdef class MyElement:
-     cdef MyRing parent
-cdef class MyRing:
-     cdef MyElement basis
-\end{verbatim}
-If you forward declare an extension type that has a base class, you
-must specify the base class in both the forward declaration and its
-subsequent definition.  For example,
-\begin{verbatim}
-    cdef class A(B)
-    ...
-    cdef class A(B):
-        # attributes and methods
-\end{verbatim}
-\subsection{Using cdef Module-Level Functions in Other Modules}
-Suppose there is a cdef function in a Cython file \code{A.pyx} that is
-not a method of a class, but you would like to use it from another
-Cython module.  There are (at least) two options.
-One option is to make the functions methods of a cdef'd
-class, then instantiate an object of that type in the other file
-(this has no weird linking issues).
-Another is to use ``public extern'' and link in the whole module, which we describe
-below.
-For example, the file \code{A.pyx} might contain
-\begin{verbatim}
-cdef int sqr_int(int x):
-    return x*x
-\end{verbatim}
-You would like to call this C-level method directly from a different Cython module.
-To do so, {\em it is crucial} to declare the method public instead:
-\begin{verbatim}
-cdef public int sqr_int(int x):
-    return x*x
-\end{verbatim}
-Then create a file \code{A.pxd} that contains
-\begin{verbatim}
-cdef extern int sqr_int(int x)
-\end{verbatim}
-Finally, to use \code{sqr_int} in another Cython module \code{B.pyx},
-do the following:
-\begin{verbatim}
-cimport sage.rings.A     # this is just an example absolute path
-def myfunction():
-    print sage.rings.A.sqr_int(5)
-\end{verbatim}
-Any time you access one Cython module from another one,
-it's necessary that the compiler know to link in the other Cython
-module (\code{A.pyx} in our example).  Tell it to link
-in \code{A.pyx} by including \code{A.pyx}
-in the corresponding entry in \code{setup.py}.  That
-entry should look something like this:
-\begin{verbatim}
-    Extension('sage.modular.B',
-              ['sage/modular/B.pyx', 'sage/rings/A.pyx'])
-\end{verbatim}
-\note{Linking as above means an entire copy of all the code in \code{A.pyx}
-is compiled into the module for \code{B.pyx}, so \code{B.so} will be larger,
-and the C-level function you're accessing will {\em not} have access
-to any module-scope variables in \code{A.pyx}.  This can lead to weird
-core dumps, depending on the nature of your function.}
-Cython will generate a file \code{A.pxi} that also declares foo.  You
-can ignore that file.
-{\bf NOTE:} Alternatively, you could put \code{include "path/to/A.pxi"}
-in \code{B.pyx} and and directly use \code{sqr_int} in \code{B.pyx} with
-no scoping:
-\begin{verbatim}
-def myfunction():
-    print sqr_int(5)
-\end{verbatim}
-This is not recommended because it is confusing and
-unpythonic to not have the scoping information.
-\subsection{Making it possible to create weak references to extension classes}
-By default, extension types do not support having weak references made
-to them. You can enable weak referencing by declaring a C attribute of
-type object called \code{__weakref__}. For example,
-\begin{verbatim}
-cdef class MyRing:
-   """
-   There is supposed to be at most one instance of this ring, so
-   we keep a weak reference pointer to it, so it will be deleted
-   from memory when there are no more user references to it.
-   """
-    cdef object __weakref__
-\end{verbatim}
-\subsection{Public and Private Variables}
-To  expose a variable to external Python, use \code{cdef public}, e.g.,
-\begin{verbatim}
-cdef class MyRing:
-    cdef public Integer characteristic
-\end{verbatim}
-Use readonly if you want to enable only readonly access.
-\begin{verbatim}
-cdef class MyRing:
-    cdef readonly Integer characteristic
-\end{verbatim}
-\subsection{Arbitrary Attributes}
-Reminder: Like builtin types, Cython extension types don't have a
-\code{__dict__} field, so you cannot assign arbitrary attributes to
-them.    That said, it is just a few lines to implement attribute
-assignment, as illustrated by the following notebook session:
-\begin{verbatim}
-{{{
-%cython
-cdef class X:
-    cdef object __dict__
-    def __init__(self):
-        self.__dict__ = {}
-    def __setattr__(self, attr, val):
-        self.__dict__[attr] = val
-    def __getattr__(self, attr):
-        try:
-            return self.__dict__[attr]
-        except KeyError:
-            raise AttributeError, "object has no attribute '%s'"%attr
-///
-}}}
-{{{
-x = X()
-///
-}}}
-{{{
-x.a = 5
-///
-}}}
-{{{
-x.a
-///
-}}}
-{{{
-x.b
-///
-Traceback (most recent call last):    x.b
-  File "/Volumes/HOME/sage-stable/local/lib/python2.5/", line 1, in <module>
-  File "/Volumes/HOME/.sage//spyx/sage20/sage20_0.pyx", line 14, in sage20_0.X.__getattr__
-    raise AttributeError, "object has no attribute '%s'"%attr
-AttributeError: object has no attribute 'b'
-}}}
-\end{verbatim}
-\subsection{Special Methods}
-Certain special methods {\em must} be defined in every extension class
-in order to work.  In particular, they do not derive from higher classes.
-This can be very confusing if you are not aware of it.  For
-example, suppose you define two classes $A$ and $B$ as follows:
-\begin{verbatim}
-cdef class A:
-   def __hash__(self):
-      return 0
-cdef class B(A):
-   def __repr__(self):
-      return "I am class B"
-\end{verbatim}
-Then $B$ will {\em not} inherit the \code{__hash__} method
-from class $A$.  If you want $B$ to be hashable, you {\em must}
-define a hash method explicitly for $B$, e.g.,
-\begin{verbatim}
-cdef class B(A):
-   def __hash__(self):
-      return A.__hash__(B)
-   def __repr__(self):
-      return "I am class B"
-\end{verbatim}
-I do not know exactly which special methods do not get inherited.
-Definitely \code{__hash__} and \code{__richcmp__} do not.
-%\section{Gotchas}
-%\subsection{How C extension classes different from Python classes}
-%\subsection{When to use {\tt cdef class} versuses {\tt def class}
-%in Cython}
-%def allows multiple inheritance, but no c-level data structures.
-%Maybe creation is faster.   What exactly is the deal here?!
-%\subsection{\SAGE's Pyrex is Patched}
-%[[Record exactly how and why here.  Post this to the
-%mailing list every once in a while. ]]
-%\subsection{There is no +=}
-%[[solution make a C macro.  need a sage.pxi that has all special
-%convenience code like this in it, and is automatically included.]]
-%\subsection{Notation for C for Loops}
-%\subsection{Typos in Function Names and Variables}
-\subsection{Doctests}
-If \code{x.pyx} is a Cython file, then you can do
-\begin{verbatim}
-   sage -t x.pyx
-\end{verbatim}
-to test all the examples in the documentation strings
-in \code{x.pyx}.
-\subsection{Signal Handling}
-Suppose you have a Cython function like
-\begin{verbatim}
-def foo():
-    cdef int n
-    # Some long running code that gets compiled to pure C, e.g.,
-    while True:
-        n = n + 1
-\end{verbatim}
-If you call foo from the \SAGE interpreter, then when \SAGE
-gets to the ``chunk of long-running C code'' pressing control-C
-will not interrupt the calculation.  The only way to interrupt
-the calculation is to completely kill your running copy of
-\SAGE.  What a wasteful pain---surely there is a better way!
-Indeed, if you simply rewrite the above code as follows, then
-suddenly control-C will work fine!
-\begin{verbatim}
-def foo():
-    cdef int n
-    _sig_on
-    while True:
-        n = n + 1
-    _sig_off
-\end{verbatim}
-\note{In a .pyx file which will be part of the \SAGE library,
-you have to explicitly put something like
-\code{include "../ext/interrupt.pxi"} in your file,
-where you give in quotes the relative directory
-to \code{SAGE_ROOT/devel/sage/sage/ext/interrupt.pxi}.}
-In addition to SIGINT, any code wrapped in \code{_sig_on} and
-\code{_sig_off} also traps SIGABRT, SIGALRM, SIGSEV, and SIGFPE.
-{\bf IMPORTANT NOTES:}
-\begin{enumerate}
-     \item These {\em must} always come in pairs. E.g., if you have just
-          a \code{_sig_off} without a corresponding \code{_sig_on}, then control-C
-          later in the interpreter will segfault!
-        \item Do {\em not} put these in the \code{__init__} method of
-          a Cython extension class, or you'll get crashes.
-        \item Do not put these around any Cython code that calls into
-          the Python API (i.e., the generated C code should have no
-          calls to the Python library)!!
-      \item If you need to do a check of control-C inside a loop, e.g.,
-          when constructing a matrix, put \code{_sig_check} instead of using
-          \code{_sig_on} then \code{_sig_off}.
-\end{enumerate}
-\subsection{Introspection}
-Users currently can't view the source code of functions
-from the command line.  This must change.  Ideas for fixing
-this problem.
-If we can find the module name where foo is defined,
-then check if the appropriate foo.pyx file is available.
-Load it.  Find the function.  Figure out which if it
-is defined twice.  Show all if we can't figure it out.
-If totally screwed and we can't even find the file, then
-modify pyrex so it embeds extra info in the docstring.
-Strip that out on displaying.
-%\section{Automatic Coercion to and From C Data Types}
-%Integer literals
-%type checking
-%Etc.
-%How to do explicit casts
-\subsection{Faster Iteration through a Python List or Tuple}
-For example, define a function \code{mysum} in Cython as
-follows.  Note the trick of void
-pointers to avoid Cython reference counting, etc:
-\begin{verbatim}
-def mysum(x):
-    cdef object seq
-    cdef int len,i
-    cdef object item
-    cdef object total
-    total = 0
-    cdef PyObject** X
-    X = FAST_SEQ_UNSAFE(x)
-    for i from 0 <=i<len:
-        item = <object>X[i]
-        total = total + item
-    return (total)
-\end{verbatim}
-The call to \code{FAST_SEQ_UNSAFE} gets direct very very
-unsafe access to the underlying array of PyObject's.
-Somewhat surprisingly, this is actually {\em faster}
-than Python's built in sum function:
-%skip
-\begin{verbatim}
-sage: time sum(range(5*10^6))
-CPU times: user 1.59 s, sys: 0.08 s, total: 1.67 s
-12499997500000L
-sage: time mysum(range(5*10^6))
-CPU times: user 1.46 s, sys: 0.03 s, total: 1.49 s
-12499997500000L
-\end{verbatim}
-In the next example, we illustrate the above idea by defining
-several simplified versions of the \code{range} command.
-\begin{verbatim}%skip
-%cython
-def f0(int n):
-  return eval('[%s for _ in xrange(%s)]'%(n, n))
-def f1(int n):
-  v = []
-  for i from 0 <= i < n:
-      v.append(i)
-  return v
-def f2(int n):
-  cdef int i
-  v = [None] * n
-  for i from 0 <= i < n:
-      v[i] = i
-  return v
-def f3(int n):
-  cdef int i
-  w = [None] * n
-  cdef void** X
-  X = FAST_SEQ_UNSAFE(w)
-  for i from 0 <= i < n:
-      Py_DECREF(<PyObject*>X[i])
-      X[i] = PyInt_FromLong(i)
-  return w
-def f4(int n):
-  cdef int i
-  v = PyList_New(n)
-  for i from 0 <= i < n:
-      PyList_SET_ITEM(v, i, <object>PyInt_FromLong(i))
-      #WARNING -- maybe I need to incref the ref to the long?!
-  return v
-\end{verbatim}
-The fourth one is the fastest.
-\subsection{getslice}
-For fastest speed, use this declaration:
-\begin{verbatim}
-def __getslice__(self, Py_ssize_t i, Py_ssize_t j):
-    ...
-\end{verbatim}
-%\section{The Python/C API}
-% The {\tt object} type.
-%\section{C Header Files}
-%\section{Writing Your Code in a Mix of Cython and C/C++}
-%\section{Splitting Implementations Across Multiple Files}
-%\section{Wrapping C++ Code}
-%Cython code compiles with C++ compiler.  Using pyrexembed.
-%\section{Other Optimization Tricks}
-%\section{Misc}
-%\subsection{Class methods}
-%In a cdef'd class this is not possible at all.
-\subsection{List Comprehensions and In-Place operators}
-List comprehensions and in-place operators are fully supported in Cython.
-(They are not supported in Pyrex, but Robert Bradshaw added support
-for them to Cython.)
-\subsection{Cython: Dealing with weird build errors}
-\begin{itemize}
-\item {\bf Problem:} You get an error like this
-\begin{verbatim}%skip
-...
-lude/python2.5 -c sage/modules/complex_double_vector.c
-      -o build/temp.macosx-10.3-i386-2.5/sage/modules/complex_double_vector.o -w
-sage/modules/complex_double_vector.c:366:
-       error: field '__pyx_base' has incomplete type
-sage/modules/complex_double_vector.c:373:
-       error: field '__pyx_base' has incomplete type
-error: command 'gcc' failed with exit status 1
-sage: There was an error installing modified sage library code.
-\end{verbatim}
-{\bf Solution:} In this case I added \code{cimport sage.structure.element} to \code{complex_double_vector.pxd}
-and the problem disappeared.  In general, look at the generated C file to see what type is incomplete (i.e.,
-not properly declared), then add an explicit cimport to your pxd file (or top of your pyx file if you
-don't have a pxd file).
-\item {\bf Problem:} Errors such as "XXX does not appear to be the correct type object".\\
-{\bf Solution:} This can happen when the build system doesn't properly detect a dependency
-and regenerate the corresponding files.  If you know which .pyx file should be recompiled,
-just touch it.  If you don't, you can always type \code{sage -ba} to brutally
-rebuild everything, which will take about 3-5 minutes.
-\end{itemize}
-%\section{To Add}
-%\begin{verbatim}
-%Parsifal Herzog wrote:
-% >     cdef va_list ap   <--- line 112
-% >     cdef voidptr oi
-% >     va_start(ap, o1)
-% >     oi = va_arg(ap, voidptr)
-% >     while oi != 0:
-% >         l.append(<object>oi)
-% >         oi = va_arg(ap, voidptr)
-% >     va_end(ap)
-% Here is an example how to use va_args:
-% http://codespeak.net/svn/lxml/trunk/src/lxml/cstd.pxd
-% The macros are defined at the the top of this file:
-% http://codespeak.net/svn/lxml/trunk/src/lxml/etree_defs.h
-% Stefan
-% _______________________________________________
-% Pyrex mailing list
-% Pyrex@lists.copyleft.no
-% http://lists.copyleft.no/mailman/listinfo/pyrex
-% \end{verbatim}
-% \begin{verbatim}
-% > Ok
-% >
-% > about my casting question. I should have declared
-% >
-% > cdef int c_length
-% >
-% > in which case the code works correctly. However in one place I had
-% > forgot the int and just typed
-% >
-% > cdef c_length
-% >
-% > and this led to the example I sent. What type of object is created if
-% > you declare with cdef but forget the type.
-% The default is a Python object.  It's equivalent to
-%     cdef object c_length.
-% William
-% \end{verbatim}
-\section{C++ Nuances}
-(This section was written by Joel B. Mohler.  Many of the tricks come from
-Martin Albrecht and David Harvey.)
-Cython can generate C++ for wrapping of C++ libraries.  There are just a
-few things to keep in mind though.  Cython doesn't know specifically about
-C++ features like function and operator overloading.  Therefore, your library
-must have unique function names or you must alias them.  Other issues
-which arise have to do with functions taking references and pointer
-dereferencing since C does not support either of these.
-You can see real live examples for many of these concepts in the NTL
-wrapper classes.  These are in the \SAGE tree at \code{sage/libs/ntl/}.
-\subsection{Aliases for Overloading (or other purposes)}
-Cython doesn't support overloaded functions, but C++ libraries readily use
-this feature.  Cython does provide a very simple way to alias these functions
-which we illustrate with an example borrowed from the NTL wrapper code.
-\begin{verbatim}
-cdef extern from "ntl_wrap.h":
-    # really, this is from NTL/ZZ.h
-    ctypedef struct ZZ_c "struct ZZ":
-        pass
-    void ZZ_add "add"( ZZ_c x, ZZ_c a, ZZ_c b)
-    #### ZZ_p_c
-    ctypedef struct ZZ_p_c "struct ZZ_p":
-        pass
-    void ZZ_p_add "add"( ZZ_p_c x, ZZ_p_c a, ZZ_p_c b)
-\end{verbatim}
-Notice that NTL supports the \code{add} function for all of its many types.
-We rename them for use in Cython as \code{<Type>_add} where \code{<Type>}
-is the NTL type name of the objects being added.
-Another type of aliasing in that example is the structure \code{ZZ} is called
-\code{ZZ_c}.  We did this simply to alleviate confusion with the \SAGE integer
-class commonly known as \code{ZZ}.  This trick is also useful if you have a
-C++ library that uses namespaces.  (Actually NTL does use namespaces,
-but we were lazy and instead inserted a \code{using namespaces NTL;} in
-a header file).
-\subsection{Pointers and References}
-Given a pointer to a C/C++ type, Cython does not support the use of '*' to
-dereference this pointer.  Here's a way around that:
-\begin{verbatim}
-    cdef Foo *t
-    t = <something>
-    function_that_takes_a_reference(t[0])
-\end{verbatim}
-In C, \code{t[0]} is the first element of the array pointed to by \code{t}.  Remember
-that C arrays are simply pointers.  Note that this trick is rarely necessary in C
-since most C code operates by passing around pointers to structures.  They are
-not ordinarily dereferenced.
-\subsection{Embedding C++ objects in Cython extension structures}
-Suppose that you have a C++ class (I'll call it Foo) which you want to wrap for
-access from python.  There would be two basic ways to go about managing
-this structure:
-\begin{itemize}
-\item Initialize a point to the object in the \code{__new__} function in my
-python/Cython object and delete in the  \code{__dealloc__}.  This method
-is pretty easy to understand from a C++ perspective.
-\item Declare the object as member of the Cython cdef'ed class.  This method
-is not quite so straightforward, but it saves you a memory allocation.  This might
-be a decent speed gain for elements of some ring where an object is allocated
-for every arithmetic operation.
-\end{itemize}
-We let the details of the first option as an exercise to the reader and provide a
-more elaborate description for the second option.  Use code like the following
-to embed the object in the Cython extension class structure:
-\begin{verbatim}
-cdef class FooWrapper:
-     cdef Foo x
-\end{verbatim}
-This may appear to work fine because many C++ constructors don't do a whole
-lot of interesting work and Cython zeroes out the memory in these structures.
-However, there is a problem since neither the constructor nor destructor are called
-for Foo since the memory is allocated with malloc (or PyMalloc or something).
- So, you need to call the constructor and destructor manually.  The rest of this
-section provides a simple way to do so.
-The main trick is that we need to call the constructor of Foo, but the memory
-is already allocated.  An important nuance of the C++ \code{new} operator is
-described here:
-\url{http://publib.boulder.ibm.com/infocenter/macxhelp/v6v81/index.jsp?topic=/com.ibm.vacpp6m.doc/language/ref/clrc05cplr199.htm}.
-There is a C++ header file in the \code{sage_c_lib} which defines some templated
-functions to make writing wrappers for C++ classes more convenient.  You can
-make the following declarations in a pxd or pxi file for use from Cython.
-\begin{verbatim}
-cdef extern from "Foo.h":
-    ctypedef struct Foo:
-        pass
-cdef extern from "ccobject.h":
-    # Some non-allocating versions
-    Foo* Foo_construct "Construct<Foo>"(void *mem)
-    void Foo_destruct "Destruct<Foo>"(Foo *mem)
-    # some allocating versions for completeness
-    Foo* Foo_new "New<Foo>"()
-    void Foo_delete "Delete<Foo>"(Foo *mem)
-\end{verbatim}
-Now, we need to call these functions we defined from our Cython
-\code{__new__} and \code{__dealloc__} functions.
-\begin{verbatim}
-cdef class FooWrapper:
-    def __init__(self, ...):
-        pass
-    # Note that the parameters to __new__ must be identical to __init__
-    # This is due to some Cython vagary
-    def __new__(self, ...):
-        Foo_construct(&self.x)
-    def __dealloc__(self):
-        Foo_destruct(&self.x)
-\end{verbatim}
-\chapter{\SAGE{} Interfaces}
-This chapter is about SAGE interfaces, both
-the C/C++ level interfaces, and the pseduo-tty interfaces,
-in \SAGE programming.
 \section{The PARI C-library Interface}
 (This chapter was written by Martin Albrecht.)
 Here is the step-by-step guide to adding a new PARI functions to \SAGE.
 We use the Frobenius form of a matrix as an example.
 The heavy lifting for matrices over integers is implemented using the
 PARI library. To compute the frobenius form in PARI the
+PARI library. To compute the Frobenius form in PARI the
 \code{matfrobenius} function is used.
 There are two ways to interact with the PARI library from \SAGE: The gp
+interface uses the gp interpreter and the PARI interface uses direct calls to
+the PARI C functions and is the preferred way as it is much faster.
+interface uses the gp interpreter, and the PARI interface uses direct calls to
+the PARI C functions --- this is the preferred way as it is much
+faster.  Thus this section focuses on using PARI.
+So we need to add a new method to the gen class which is the abstract
+representation of all PARI library objects. That means that if we add a
+method to this class every PARI object regardless if it is a number,
+polynomial or matrix will have our new method. So you can do
+\code{pari(1).matfrobenius()} but you will receive a PariError in this case.
+So we will add a new method to the gen class: this is the abstract
+representation of all PARI library objects. That means that once we add a
+method to this class, every PARI object, whether it is a number,
+polynomial or matrix, will have our new method. So you can do
+\code{pari(1).matfrobenius()}, but since PARI wants to apply
+\code{matfrobenius} to matrices, not numbers, you will receive a
+PariError in this case.
+The gen class is defined in sage/libs/pari/gen.pyx where we add the method
+\code{matfrobenius}:
+The gen class is defined in
+\code{SAGE_ROOT/devel/sage/sage/libs/pari/gen.pyx}, and this is where we add
+the method \code{matfrobenius}:
 \begin{verbatim}
 …
         return self.new_gen(matfrobenius(self.g, flag))
 \end{verbatim}
 The \verb+_sig_on+ statement is some magic to prevent SIGSEGVs from the pari C
+The \code{_sig_on} statement is some magic to prevent SIGSEGVs from the PARI C
 library to crash the \SAGE interpreter by catching these signals. The
 \verb+self.new_gen()+ call constructs a new SAGE-python-gen object from a given
+\code{self.new_gen()} call constructs a new SAGE-python-gen object from a given
 pari-C-gen where the pari-C-gen is stored as the SAGE-python-gen.g attribute.
 The matfrobenius call is just a call to the pari C library function
 "matfrobenius" with the appropriate parameters.
+The \code{matfrobenius} call is just a call to the PARI C library function
+\code{matfrobenius} with the appropriate parameters.
 The information which function to call and how to call it can be retrieved
 from the PARI users manual (note: \SAGE includes the development version of
 PARI so check the development version's user manual). Looking for
+The information about which function to call and how to call it can be retrieved
+from the PARI user's manual (note: \SAGE includes the development version of
+PARI, so check that version of the user's manual). Looking for
 \code{matfrobenius} you can find:
 \verb+"The library syntax is matfrobenius(M,flag)"+
+\verb+"The library syntax is matfrobenius(M,flag)"+.
+Please not that the pari C function may have a different name than gp function
+(see e.g. \code{mathnf}), so always check with the manual.
+In case you are familiar with gp,
+please note that the PARI C function may have a different name than
+the corresponding gp function
+(for example, see \code{mathnf}), so always check the manual.
 At this point we are done with the PARI interface and may add some more
+At this point we are done with the PARI interface and can add some more
 interfaces around it for convenience:
+First we add a functional representation of the method to
+sage/libs/pari/functional.py so we can do:
+\verb+matfrobenius(<some.pari.gen>)+ additionally to
+\verb+<some.pari.gen>.matfrobenius()+.
+first we add a functional representation of the method to
+\code{SAGE_ROOT/devel/sage/sage/libs/pari/functional.py}
+so we can do
+\code{matfrobenius(<some.pari.gen>)} as well as
+\code{<some.pari.gen>.matfrobenius()}:
 \begin{verbatim}
 def matfrobenius(self): return pari(self).matfrobenius()
 \end{verbatim}
+Then we also add a frobenius(flag) method to the matrix_integer class where we
+call the \code{matfrobenius()} method on the \verb+_pari_()+ object associated with the
+matrix after doing some sanity checking. Then we need to convert the pari gen
+to some meaningful \SAGE objects depending on the return value as described in
+the PARI user's manual.
+Then we also add a \code{frobenius(flag)} method to the
+\code{matrix_integer} class where we call the \code{matfrobenius()}
+method on the PARI object associated to the matrix after doing some
+sanity checking. Then we convert output from PARI to \SAGE objects:
 \begin{verbatim}
     def frobenius(self,flag=0):
         """
+        Return the Frobenius form of this matrix.
+        If flag is 0 (the default value), return the Frobenius
+            form of this matrix.
         If flag is 1, return only the elementary divisors.
         If flag is 2, return a two-components vector [F,B]
         where F is the Frobenius form and B is the basis change
         so that M=B^-1*F*B.
+        If flag is 2, return a two-component vector [F,B]
+            where F is the Frobenius form and B is the basis change
+            so that M=B^-1*F*B.
         INPUT:
            flag -- 0,1 or 2 as described above
 …
         elif flag==1:
             r = polynomial_ring.PolynomialRing(self.base_ring())
             #BUG: this should be handled in PolynomialRing not here
+            return [eval(str(x).replace("^","**"),{},r.gens_dict()) for x in v.python_list()]
+            return [eval(str(x).replace("^","**"),{},r.gens_dict())
+                    for x in v.python_list()]
         elif flag==2:
+            F = matrix_space.MatrixSpace(rational_field.RationalField(),self.nrows())(v[0].python())
+            B = matrix_space.MatrixSpace(rational_field.RationalField(),self.nrows())(v[1].python())
+            F = matrix_space.MatrixSpace(rational_field.RationalField(),
+                                         self.nrows())(v[0].python())
+            B = matrix_space.MatrixSpace(rational_field.RationalField(),
+                                         self.nrows())(v[1].python())
             return F,B
 \end{verbatim}
-Then we add some examples, wait for make test to complete without errors and
-commit.
 \section{GAP}
 …
 Wrapping a GAP function in \SAGE is a matter of writing a program in
 Python which uses the pexpect interface to pipe various commands to
 GAP and read back the input into \SAGE.  This can range from easy to
+GAP and read back the input into \SAGE.  This is sometimes easy, sometimes
 hard.
 For example, suppose we want to make a wrapper for computation of the
+For example, suppose we want to make a wrapper for the computation of the
 Cartan matrix of a simple Lie algebra. The Cartan matrix of $G_2$ is
 available in GAP using the commands
 %skip
 …
 (Incidentally, most of the GAP Lie algebra implementation was written by
 Thomas Breuer, Willem de Graaf and Craig Struble.)
 In \SAGE, one can simply type
+In \SAGE, one can access these commands by typing
 \begin{verbatim}
 sage: L = gap.SimpleLieAlgebra('"G"', 2, 'Rationals'); L
 …
 \end{verbatim}
 Note the \code{'"G"'} which is evaluated in Gap as the string \code{"G"}.
+Using this example, the purpose of this section is
+to show how one might write a Pyhon/\SAGE program
+who's input is, say, \code{('G',2)} and who's output
+is the matrix above (but as a \SAGE Matrix - see
+\code{matrix.py} in \code{sage/matrix/matrix.py}).
+The purpose of this section is to use this example
+to show how one might write a Python/\SAGE program
+whose input is, say, \code{('G',2)} and whose output
+is the matrix above (but as a \SAGE Matrix --- see
+the code in the directory \code{SAGE_ROOT/devel/sage/sage/matrix/} and the
+corresponding parts of the \sage reference manual).
 First, the input must be converted into strings consisting of legal
 GAP commands. Then the GAP output, which is also a string, must be
 …
     L = gap.SimpleLieAlgebra('"%s"'%type, rank, 'Rationals')
     R = L.RootSystem()
-    sM  = R.CartanMatrix()
-    ans = eval(str(sM))
-    MS  = MatrixSpace(ZZ, rank)
-    return MS(ans)
-\end{verbatim}
-The output {\tt ans} is a Python list. The last two lines convert that
-list to a \SAGE class object Matrix instance.
-Alternatively, one could code the body of the above function in
-a more pythonic way as follows:
-%skip
-\begin{verbatim}
-    L = gap.new(SimpleLieAlgebra("%s", %s, Rationals);'%(type, rank))
-    R = L.RootSystem()
     sM = R.CartanMatrix()
     ans = eval(str(sM))
     MS  = MatrixSpace(QQ, rank)
     return MS(ans)
 \end{verbatim}
+The output \code{ans} is a Python list. The last two lines convert that
+list to an instance of the \sage class \code{Matrix}.
+Alternatively, one could replace the first line of the above function
+with this:
+%skip
+\begin{verbatim}
+    L = gap.new(SimpleLieAlgebra("%s", %s, Rationals);'%(type, rank))
+\end{verbatim}
 Defining ``easy'' and ``hard'' is subjective, but here is one
 definition: an example is ``easy'' if there is already a corresponding
 class object in Python or \SAGE for the output data type of the GAP
+definition: wrapping a GAP function is ``easy'' if there is already a corresponding
+class in Python or \SAGE for the output data type of the GAP
 function you are trying to wrap.  For example, wrapping any GUAVA
 (GAP's error-correcting codes package) function is ``easy'' since
 error-correcting codes are vector spaces over finite fields and GUAVA
 …
 \item[(e)]
 integers.
 \end{itemize}
 \SAGE already has a class object for each of these.
+\SAGE already has classes for each of these.
 A ``hard'' example is left as an exercise!  Here are a few ideas.
 \begin{itemize}
 \item Write a wrapper for GAP's {\tt FreeLieAlgebra} function (or,
+\item Write a wrapper for GAP's \code{FreeLieAlgebra} function (or,
   more generally, all the finitely presented Lie algebra fuunctions in
   GAP).  This would require creating new Python objects.
 \item Write a wrapper for GAP's {\tt FreeGroup} function (or, more
+\item Write a wrapper for GAP's \code{FreeGroup} function (or, more
   generally, all the finitely presented groups fuunctions in GAP).
   This would require writing some new Python objects.
 …
 \end{itemize}
 \section{Singular}
 (The first version of this chapter was written by David Joyner.)
 Using Singular functions from \SAGE is not much different conceptually than using
+GAP functions from \SAGE. This can range from very easy to hard, depending on how
+much of the data structure of the output of the Singular function is already
+present in \SAGE.
+Using Singular functions from \SAGE is not much different conceptually
+than using GAP functions from \SAGE. As with GAP, this can range from
+easy to hard, depending on how much of the data structure of the
+output of the Singular function is already present in \SAGE.
 First, some terminology. For us, a \emph{curve} $X$ over a finite field $F$ is a
+First, some terminology. For us, a \emph{curve} $X$ over a finite field $F$ is an
 equation of the form $f(x,y)=0$, where $f\in F[x,y]$ is a polynomial. It may or
 may not be singular.  A \emph{place of degree} $d$ is a Galois orbit of $d$
 points in $X(E)$, where $E/F$ is degree $d$. For example, a place of degree $1$
 …
 % }
+>From looking at the output, notice that our wrapper program will need to
+parse the string represented by $L$ above.
+Let us write a separate program to do just that. This requires figuring out how
+From looking at the output, notice that our wrapper program will need to
+parse the string represented by $L$ above, so
+let us write a separate program to do just that. This requires figuring out how
 to determine where the coordinates of the points are placed in the string L.
 Python has some very useful string manipulation commands to do just that.
 …
 Now it is an easy matter to put these ingredients together into a \SAGE function
+which takes as input a triple $(f,F,d)$: finite field $F$ \emph{of prime order},
+a polynomial $f$ in $F[x,y]$ defining $X:\  f(x,y)=0$ (note the variables $x,y$
+must be used), and the degree $d$. The output is the number of points of places
+in $X$ of degree $d=1$ over $F$. At the moment, their is no ``translation''
+which takes as input a triple $(f,F,d)$:
+a polynomial $f$ in $F[x,y]$ defining $X:\  f(x,y)=0$ (note that the variables $x,y$
+must be used),
+a finite field $F$ \emph{of prime order},
+and the degree $d$. The output is the number of places
+in $X$ of degree $d=1$ over $F$. At the moment, there is no ``translation''
 between elements of $GF(p^d)$ in Singular and \SAGE unless $d=1$. So, for this
 reason, we restrict ourselves to points of degree one.
 …
     return points_parser(L,F)
 \end{verbatim}
+The triple quoted string documentation is important! It not only helps users
+understand your function's use and syntax but also, if the function is included
+in \SAGE, will be reproduced in the printed and online documentation, and
+\emph{automatically tested} (even the output) before each \SAGE release.
+Note also that the ordering returned by this \SAGE function is exactly the same
+Note that the ordering returned by this \SAGE function is exactly the same
 as the ordering in the Singular variable \code{POINTS}.
 %\comment{
 …
 %number_of_places(f,F,d)
 %}
 Some examples:
+One more example (in addition to the one in the docstring):
 %skip
 \begin{verbatim}
-sage: F = GF(5)
-sage: R = MPolynomialRing(F,2,names = ["x","y"])
-sage: x,y = R.gens()
-sage: f = y^2-x^9-x
-sage: places_on_curve(f,F)
-      ((0, 1, 0), (3, 4, 1), (0, 0, 1), (2, 3, 1), (3, 1, 1), (2, 2, 1))
 sage: F = GF(2)
 sage: R = MPolynomialRing(F,2,names = ["x","y"])
 sage: x,y = R.gens()
 …
       ((0, 1, 0), (1, 0, 0), (0, 0, 1))
 \end{verbatim}
+\section{Another Approach}
+There is also a more python-like interface to Singular.  Using this
+\section{Singular: Another Approach}
+There is also a more Python-like interface to Singular.  Using this
 the code is much simpler, as illustrated below.   First we demonstrate
 computing the places on a curve in a particular case.
 …
 sage: [(L[i][1], L[i][2], L[i][3]) for i in range(1,7)]
       [(0, 1, 0), (-2, 1, 1), (0, 0, 1), (2, 2, 1), (-2, -1, 1), (2, -2, 1)]
 \end{verbatim}
-The \code{name} method of a Singular object returns the name
-of that object in the Singular interpreter, so that it can
-be used as input to a Singular function.
 Next we implement the general function (we omit the docstring, which
+Next we implement the general function (for brevity we omit the docstring, which
 is the same as above).  Note that the \code{point_parser} function
 is not required.
 …
 used only the barest minimum of that interface.
 \section{Creating a new Pseudo-tty Interface}
+\SAGE pseudo-tty interfaces can be created that allow \SAGE to
+work with an almost completely arbitrary command line program,
+and don't require any modification or extensions to the
+command line program.   They are also {\em surprisingly} fast
+You can create \SAGE pseudo-tty interfaces that allow \SAGE to
+work with almost any command-line program,
+and which don't require any modification or extensions to that
+program.  They are also {\em surprisingly} fast
 and flexible (given how they work!), because all IO is
 buffered, and because interaction between \sage and the
+the command line program can be non-blocking (asynchronous),
+using the \code{_send}, \code{_so_far}, and \code{_get}
+methods.
+command line program can be non-blocking (asynchronous); this is
+because they all derive from the \sage class \code{Expect}, which
+handles the communication between \sage and the external process.
+% To create a new interface to a computer algebra system,
+% do the following:
+% \begin{enumerate}
+% \item Copy
+For example, here is part of the file
+\code{SAGE_ROOT/devel/sage/sage/interfaces/octave.py}, which defines
+an interface between \sage and Octave, an open-source program
+for doing numerical computations, among other things.
+%skip
+\begin{verbatim}
+import os
+from expect import Expect, ExpectElement
+% \section{Projects}
+% For the interested reader, here are a few ideas for some other wrappers,
+% which are not yet implemented in \SAGE.
+class Octave(Expect):
+\end{verbatim}
+The first two lines import the library \code{os}, which contains
+operating system routines, and also class \code{Expect}, which is the
+basic class for interfaces.  The third line defines the class
+\code{Octave}: it derives from \code{Expect}.  After this comes a
+docstring, which we omit here -- see the file for details.  Next comes:
+%skip
+\begin{verbatim}
+    def __init__(self, maxread=100, script_subdirectory="", logfile=None,
+                 server=None, server_tmpdir=None):
+        Expect.__init__(self,
+                        name = 'octave',
+                        prompt = '>',
+                        command = "octave --no-line-editing --silent",
+                        maxread = maxread,
+                        server = server,
+                        server_tmpdir = server_tmpdir,
+                        script_subdirectory = script_subdirectory,
+                        restart_on_ctrlc = False,
+                        verbose_start = False,
+                        logfile = logfile,
+                        eval_using_file_cutoff=100)
+\end{verbatim}
+This uses the class \code{Expect} to set up the Octave interface.
+%skip
+\begin{verbatim}
+    def set(self, var, value):
+        """
+        Set the variable var to the given value.
+        """
+        cmd = '%s=%s;'%(var,value)
+        out = self.eval(cmd)
+        if out.find("error") != -1:
+            raise TypeError, "Error executing code in Octave\nCODE:\n\t%s\nOctave ERROR:\n\t%s"%(cmd, out)
+% \begin{itemize}
+    def get(self, var):
+        """
+        Get the value of the variable var.
+        """
+        s = self.eval('%s'%var)
+        i = s.find('=')
+        return s[i+1:]
+% \item
+% Write a \SAGE function, possibly containing Singular code,
+% which ``translates'' the Singular generator
+% \code{a} of $\GF(p^d)$ to a \SAGE generator of $\GF(p^d)$.
+    def console(self):
+        octave_console()
+\end{verbatim}
+These let users type \code{octave.set('x', 3)}, after which
+\code{octave.get('x')} returns \code{' 3'}.  Running
+\code{octave.console()} dumps the user into Octave interactive shell.
+%skip
+\begin{verbatim}
+    def solve_linear_system(self, A, b):
+        """
+        Use octave to compute a solution x to A*x = b, as a list.
+% \item
+% Write a wrapper for Singular's
+% \code{BrillNoether} function to compute bases of Riemann-Roch spaces.
+% This would not require writing some new Python objects.
+        INPUT:
+            A -- mxn matrix A with entries in QQ or RR
+            b -- m-vector b entries in QQ or RR (resp)
+% \item
+% Write a wrapper for Singular's
+% \code{groebner} function to compute Groebner bases.
+% This would not require writing some new Python objects.
+        OUTPUT:
+            An list x (if it exists) which solves M*x = b
+% \item
+% Write a wrapper for Singular's method for
+% ``Solving systems of polynomial equations'' in
+% \url{http://www.singular.uni-kl.de/Manual/3-0-0/sing_502.htm\#SEC554}
+        EXAMPLES:
+            sage: M33 = MatrixSpace(QQ,3,3)
+            sage: A   = M33([1,2,3,4,5,6,7,8,0])
+            sage: V3  = VectorSpace(QQ,3)
+            sage: b   = V3([1,2,3])
+            sage: octave.solve_linear_system(A,b)    # requires optional octave
+            [-0.33333299999999999, 0.66666700000000001, -3.5236600000000002e-18]
+% \item
+% Write a wrapper for Singular's method for constructing and decoding
+% ``AG codes'' in
+% \url{http://www.singular.uni-kl.de/Manual/3-0-0/sing_503.htm\#SEC555}
+% \end{itemize}
+        AUTHOR: David Joyner and William Stein
+        """
+        m = A.nrows()
+        n = A.ncols()
+        if m != len(b):
+            raise ValueError, "dimensions of A and b must be compatible"
+        from sage.matrix.all import MatrixSpace
+        from sage.rings.all import QQ
+        MS = MatrixSpace(QQ,m,1)
+        b  = MS(list(b)) # converted b to a "column vector"
+        sA = self.sage2octave_matrix_string(A)
+        sb = self.sage2octave_matrix_string(b)
+        self.eval("a = " + sA )
+        self.eval("b = " + sb )
+        soln = octave.eval("c = a \\ b")
+        soln = soln.replace("\n\n ","[")
+        soln = soln.replace("\n\n","]")
+        soln = soln.replace("\n",",")
+        sol  = soln[3:]
+        return eval(sol)
+\end{verbatim}
+This code defines the method \code{solve_linear_system}, which works
+as documented.
+These are only excerpts from \code{octave.py}; check that file for
+more definitions and examples.  Look at other files in the directory
+\code{SAGE_ROOT/devel/sage/sage/interfaces/} for examples of
+interfaces to other software packages.
+\chapter{Mercurial: The \sage Source Control System}
+\chapter{The \SAGE Manuals}
+\label{ch:manuals}
+This chapter describes how to modify the \SAGE manuals.
+\sage's manuals are written in \Latex, and to edit them, you just need
+to edit the appropriate file.
+If, for example, you want to change the \sage tutorial, then you
+should start by modifying the file
+\code{SAGE_ROOT/devel/doc/tut/tut.tex}.  Then to build a dvi or pdf file
+with your changes, type \code{./build_dvi} or
+\code{./build_pdf} in the \code{devel/doc/tut} directory.  You'll get
+a file \code{tut.dvi} or \code{tut.pdf}, which you should inspect.
+You should also run \code{sage -t tut.tex} to test all of the examples
+-- see Section~\ref{ch:testing}.
+You can build an HTML
+version of the tutorial by typing \code{make tut} in the
+\code{devel/doc} directory.  Once you've done this, you can access the
+new HTML version from the notebook interface to \sage by clicking the
+\code{Help} link.
+Finally, you might want to share your changes with the \sage
+community.  To do this, use Mercurial (see Chapter~\ref{ch:mercurial})
+to produce patch files, and submit them to the \sage trac server.
+The \sage manuals and the corresponding files to edit:
+\begin{itemize}
+\item The \sage tutorial:
+\code{SAGE_ROOT/devel/doc/tut/tut.tex}
+\item The \sage programming guide:
+\code{SAGE_ROOT/devel/doc/prog/prog.tex}
+\item Constructions in \sage:
+\code{SAGE_ROOT/devel/doc/const/const.tex}
+\item The \sage installation guide:
+\code{SAGE_ROOT/devel/doc/inst/inst.tex}
+\item The \sage reference manual: some of this is contained in the file
+\code{SAGE_ROOT/devel/doc/ref/ref.tex}, but most of it is
+automatically generated from the \sage source code.  See the file
+\code{SAGE_ROOT/devel/doc/ref/README.txt} for information about how to
+edit the reference manual.
+\end{itemize}
+\part{Disseminating Code for \sage}
+\chapter{Introduction}
+Whether you've developed some new code for \sage or just have a simple
+bug fix, you need to know how to communicate what you've done to other
+\sage users.  This part of the guide discusses this issue.  Here are
+some of the available avenues of communication and tools to aid in
+that communication:
+\begin{itemize}
+\item the Google group \code{sage-support}, at
+\url{http://groups.google.com/group/sage-support}.  According to the
+group's official description, ``This group gives help and support to
+those who have problems with \sage (installation, syntax, etc).''
+\item the Google group \code{sage-devel}, at
+\url{http://groups.google.com/group/sage-devel}.  According to the
+group's description, ``This list is for \sage development and is about
+programming, design and technical issues.''  This is a great place to
+post questions about whether certain behavior is a bug, or whether a
+certain feature ought to be implemented (or how it ought to be
+implemented), or similar issues.
+\item Mercurial: this is the source control system that is included
+with \SAGE.  Use this to produce patches for \sage.  See
+Chapter~\ref{ch:mercurial} for a full discussion of Mercurial.
+\item the \sage trac server, at
+\url{http://trac.sagemath.org/sage_trac/}: this is where you should
+post bugs, patches for bugs, additions to the \sage library, etc.  See
+Chapter~\ref{ch:trac} for more information.
+\end{itemize}
+\section{Inclusion Procedure for New \sage Code}
+For code to become part of \sage's core, it must meet the
+following requirements:
+\begin{itemize}
+\item \textbf{License}. The license must be a GPL version 2+
+compatible license. (This will be publicly revisited around Jan 15,
+.)
+\item \textbf{Build Support}.
+The code must build on our supported architectures and compilers
+(and intended port targets):
+\begin{itemize}
+\item Linux: x86, x86_64, Itanium, ppc, ppc64, Sparc (gcc 3.4-4.3)
+\item Apple Mac OS X: ppc, ppc64, x86, x86-64 (Xcode 2.5+)
+\item Microsoft Windows: x86, x86_64 MSVC 2005/Intel Fortran (MinGW or
+Cygwin support is insufficient!)
+\item Solaris 10: Sparc, x86, x86_64 (Sun Forte 12)
+\end{itemize}
+Remarks:
+\begin{itemize}
+\item Some \sage developers are willing to help you port to OSX,
+Solaris and Windows. But this is no guarantee and you or your project
+are expected to do the heavy lifting and also support those ports
+upstream if there is no \sage developer who is willing to share the
+burden.
+\end{itemize}
+Potential future ports include FreeBSD (x86, x86-64), OpenBSD (x86,
+x86-64), HPUX (Itanium), AIX (PPC64), and ARM (OSX).
+\item \textbf{Quality}.  The code should be ``better'' than any other
+available code (that passes the two above criteria), and the authors
+need to justify this. The comparison should be made to both Python and
+other software. Criteria in passing the quality test include:
+\begin{itemize}
+\item Speed
+\item Documentation
+\item Usability
+\item Memory leaks
+\item Maintainable
+\item Reasonable build time, size, dependencies
+\end{itemize}
+\item \textbf{Refereeing}.  The code must be refereed, as discussed in
+Chapter~\ref{ch:trac}.
+\end{itemize}
+\chapter{Producing Patches with Mercurial}
 \label{ch:mercurial}
+Mercurial is the official source control system that is included with
+\SAGE.  Mercurial is robust and works well even with huge data sets.
+It's a solid piece of quality software.
+    {\it All} the Mercurial repositories related to \SAGE are included
+    with \SAGE by default.  Thus the complete change history and setup
+    for doing development is available in your copy of \SAGE by
+    default.  It's not something that has to be confusingly installed
+    later.   \SAGE is designed to be extremely developer friendly.
+If you are editing or adding to \sage's core library, you will
+probably want to share your changes with other users.  Mercurial is
+the tool to do this.  Mercurial is the source control system that is
+included with \SAGE.  This chapter provides an overview of how to use
+Mercurial with \sage; see \url{http://www.selenic.com/mercurial/} for
+full documentation on Mercurial.
+\SAGE includes by default these Mercurial repositories:
+All of the Mercurial repositories related to \SAGE are included
+with \SAGE.  Thus the complete change history and setup
+for doing development is available in your copy of \SAGE.
+Before using Mercurial,
+make sure to define your username and password so the
+patches you make are identified as yours.  Make a
+file \verb+~/.hgrc+ in your home directory like this one:
+%skip
+\begin{verbatim}
+[ui]
+username = Cardinal Fang <fang@spanish-inquisition.com>
+\end{verbatim}
+\section{Quick Mercurial Tutorial for \SAGE}
+There are several ways to run Mercurial: from the command line, run
+\code{sage -hg} (\code{Hg} is the chemical symbol for mercury), or
+from within \sage, run \code{hg_sage}.  Most of the examples below use
+the second method.
+Before you modify \sage library files, you might want to create a copy
+of the \sage library in which to work.  Do this by typing \code{sage
+-clone myver}, for example; then \SAGE will use Mercurial to clone the
+current repository and call the result \code{myver}. The new
+repository is stored in \code{<SAGE_ROOT>/devel/sage-myver}, and when
+you clone, the symlink \verb+sage --> sage-myver+ is made.
+(You can also do, e.g., \code{sage -clone -r 1250 oldver}, to get a
+clone of \SAGE but as it was at revision 1250.  Of course, dependency
+issues could make old versions not work (e.g., maybe an old \SAGE
+library won't compile with the latest Singular library, which is what
+is installed elsewhere in SAGE_ROOT).  From with \sage, type
+\code{hg_sage.log()} to see the revision history.  \note{If you clone
+an old version, all of the Cython code is rebuilt, since there is no
+easy way to know which files do and don't need rebuilding.})
+Once you've copied the library to a new branch \code{myver} and edited
+some files there, you should build the \sage library to incorporate
+those changes: type \code{sage -b myver}, or just \code{sage -b} if
+the branch \code{myver} is already the current branch: that is, if
+\code{<SAGE_ROOT>/devel/sage} links to
+\code{<SAGE_ROOT>/devel/sage-myver}.  You can also type \code{sage -br
+myver} to build the library and then to immediately run \sage.
+If you want to submit your changes to the \sage development team for
+refereeing (and inclusion into \sage if the referee's report is
+positive), you should produce patch files.  To do this:
 \begin{itemize}
+\item {\tt SAGE_ROOT/devel/sage-*} -- the \SAGE library source code
+\item {\tt SAGE_ROOT/devel/doc-*} -- the \SAGE documentation
+\item {\tt SAGE_ROOT/data/extcode} -- new code included with \SAGE that is written for the systems \SAGE interfaces with, e.g., GAP, PARI, etc.
+\item {\tt SAGE_ROOT/local/bin} -- the \SAGE shell scripts
+\item Type \code{hg_sage.status()} and \code{hg_sage.diff()} to see
+exactly what you've done (you can pass options to \code{diff} to see
+information about certain files).
+\item If you've added new files, not just edited existing ones, type
+\code{hg_sage.add([filenames])} to add those new files to your
+repository.
+\item Commit your changes by typing \code{hg_sage.commit([optional
+filenames])} to commit the changes in files to the repository --- if no
+filenames are given, all files are committed. First the output of
+\code{hg diff} is displayed: look at it or just enter \code{q}. Then
+you are dumped into an editor to type a brief comment on the changes.
+The default editor is vi, so type \code{i}, write some meaningful one
+line description, hit \code{Escape} and type \code{:wq}.  (In bash, to
+make emacs the default editor, type \code{export EDITOR=emacs}.)
+\item Now create a patch file using \code{hg_sage.export(...)}. This
+command needs a revision number (or list of revision numbers) as an
+argument; use \code{hg_sage.log()} to see these numbers. An optional
+second argument to \code{hg_sage.export(...)}  is a filename for the
+patch; the default is \code{(changeset_revision_number).patch}.
+\item Then post your patch on the \sage trac server: see
+Chapter~\ref{ch:trac}.
 \end{itemize}
+Make sure to define your username and password so the
+patches you make are identified as yours.  Make a
+file \verb+~/.hgrc+ in your home directory like this one:
+Note that you can also start a very nice web server that allows you to
+navigate your repository with a web browser, or pull patches from it
+remotely, by typing \code{hg_sage.serve()}.
+Finally, if you want to apply a patch file (perhaps you've downloaded
+a patch from the trac server for review), use the command
+\code{hg_sage.patch('filename')}.
+\section{Using Mercurial with Other \sage Repositories}
+\SAGE includes these Mercurial repositories:
+\begin{itemize}
+\item {\tt SAGE_ROOT/devel/sage-*} : the \SAGE library source code
+\item {\tt SAGE_ROOT/devel/doc-*} : the \SAGE documentation
+\item {\tt SAGE_ROOT/data/extcode} : external system code, i.e., code
+included with \SAGE that is written for the systems with which \SAGE
+interfaces, e.g., GAP, PARI, etc.
+\item {\tt SAGE_ROOT/local/bin} : the \SAGE shell scripts
+\end{itemize}
+The previous section discussed using Mercurial with the \sage
+library, via the command \code{hg_sage}.  There are corresponding
+commands for each of the repositories:
+\begin{itemize}
+\item use \code{hg_sage} for the \sage library
+\item use \code{hg_doc} for the \sage documentation
+\item use \code{hg_extcode} for the external system code
+\item use \code{hg_scripts} for the \sage shell scripts
+\end{itemize}
+To produce patches for \sage's documentation, for example, use the
+same method as outlined above, but with commands like
+\code{hg_doc.status()}, \code{hg_doc.commit()}, etc.
+\chapter{Producing New \sage Packages}
+\label{ch:spkg}
+If you are producing code to add new functionality to \sage, you might
+consider turning it into a package (an \code{spkg} file) instead of a
+patch file.  If your code is very large (for instance) and should be
+offered as an optional download, a package is the right choice;
+similarly, if your code depends on some other optional component of
+\sage, you should produce a package.
+If you're not sure whether to build an spkg file or a patch file, ask
+for advice on \code{sage-devel}.
+\section{Creating a New spkg File}
+\sage packages are distributed as .spkg files, but an .spkg file is
+just a .tar.bz2 file (or a tar file), but named with .spkg to
+discourage confusion.  In particular, you can type
+%skip
 \begin{verbatim}
+[ui]
+username = William Stein <wstein@gmail.com>
+     tar jxvf mypackage-version.spkg
+\end{verbatim}
+to extract one and see what is inside.
+Here's how to make your own spkg file:
+\begin{itemize}
+\item[(a)]
+Make a directory, e.g., \code{mypackage-0.1}.
+The name of the directory should be a lower-case string with no
+dashes, followed by a dash, followed by a version number.
+\item[(b)]
+Put your files in that directory.
+\item[(c)]
+Create an executable shell script \code{mypackage-0.1/spkg-install}.
+\item[(d)]
+The script \code{spkg-install} is run during installation
+of the \sage package.   In this script you may
+make the following assumptions:
+\begin{itemize}
+\item The PATH has the locations of \code{sage} and \code{python}
+(from the \sage installation) at the front.  Thus the command
+%skip
+\begin{verbatim}
+     python setup.py install
+\end{verbatim}
+will run the correct version of Python with everything set up
+correctly.  Also, running \code{gap} or \code{Singular}, for example,
+will run the correct version.
+\item
+The environment variable \code{SAGE_ROOT} points to
+the root directory of the \sage installation.
+\item
+The environment variable \code{SAGE_LOCAL} points to
+the \code{SAGE_ROOT/local} directory of the \sage installation.
+\item
+The environment variables
+\code{LD_LIBRARY_PATH} and \code{DYLD_LIBRARY_PATH}
+both have \code{SAGE_ROOT/local/lib} at the front.
+\end{itemize}
+\item[(e)]
+The \code{spkg-install} script should copy your files to the appropriate
+place after doing any build that is necessary.  That's it!
+\item[(f)] (Optional) Post a copy on the \sage trac server
+(Chapter~\ref{ch:trac}) so it can be refereed; if it gets a positive
+review, it might be included into the core \sage library, or it might
+become an optional download from the \sage web site, so anybody can
+automatically install it by typing \code{sage -i mypackage-version.spkg}.
+\end{itemize}
+If your package depends on another package, say GAP, then you should
+check that this other package has been installed.  In the case of GAP,
+it is installed into the directory \code{SAGE_ROOT/local/lib/gap-4.4.10/}
+and your \code{spkg-install} script should check that this exists,
+with the code like the following:
+%skip
+\begin{verbatim}
+if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.10 ]; then
+    echo "This package requires that GAP 4.4.10 is installed."
+    exit 1
+fi
 \end{verbatim}
+\section{Quick Mercurial tutorial for \SAGE}
+Mercurial is available by default since \SAGE version 1.3.7.3.   It's confusingly
+called {\tt hg} (the chemical symbol for mercury) at the command line.
+Do "sage -hg" to run it, or make a link to \verb+SAGE_ROOT/local/bin/hg+
+      The file \verb+<SAGE_ROOT>/local/lib/python/site-packages/sage+
+      is just a symbolic link to
+\emph{Caveat}: Do {\em not} just
+copy to \code{SAGE_ROOT/local/lib/gap*/} since that will copy
+your package to the lib directory of the {\em old} version of GAP if
+GAP is upgraded.
+          \verb+<SAGE_ROOT>/devel/sage/build/sage+
+External Magma code goes in
+\verb+SAGE_ROOT/data/extcode/magma/user+, so if you want to redistribute
+Magma code with \sage as a package that Magma-enabled users can use,
+that's where you'd put it.  You'd also want to have relevant Python code
+to make the Magma code easily usable.
+      Thus by changing what\verb+ <SAGE_ROOT>/devel/sage+ points to,
+      you can easily switch between running different versions of the
+      \SAGE library.  The "sage -b" command changes this symbolic link
+      for you automatically. Without argument "sage -b" switches to
+      where the symbolic link \verb+sage+ points to and builds \SAGE
+      in this branch.
+\textbf{Example}:
+These instructions provide directions for creating a particular
+\sage package.
-      If you type \code{sage -clone myver}, say, then \SAGE will use
-      hg to clone the current repository and call the result myver.
-      The new repository is stored in \verb+<SAGE_ROOT>/devel/sage-myver+
-      and when you clone the symlink \verb+sage --> sage-myver+
-      is made.  Creating clones of a repository should be fairly fast,
-      e.g., about 30 seconds.
-      You can also do, e.g.,
-                 \verb+sage -clone -r 1250 oldver+
-      to get a clone of \SAGE but as it was at revision 1250.   Of course,
-      dependency issues could make old versions not work (e.g., maybe an
-      old \SAGE library won't compile with the latest Singular library, which
-      is what is installed elsewhere in SAGE_ROOT).
-      Type \code{hg_sage.log()} to see
-      the revision history.
-      \note{All the cython code is rebuilt
-      if you clone an old version, since there is no easy way to
-      know which files do and don't need rebuilding.}
-      To switch to a different repository and build it, type, e.g.,
-                   \verb+sage -b myver+
-      if the repository is stored in
-               \verb+<SAGE_ROOT>/devel/sage-myver+
-      To switch back type \verb+sage -b main+ to the repository stored in
-      \verb+<SAGE_ROOT>/devel/sage-main+
-You can also use \code{hg} directly from the command line
-instead of using it via the \code{hg_sage} object.
-Use \code{sage -hg [options]} from within the repository.
-See \code{hg(1)} in the man pages for option details.
-If you want to do a bunch of development, then make
-it available to others, here is how:
 \begin{itemize}
 \item
+Make a new branch (using \code{sage -clone myver}).
+Download \code{guava3.6.tar.gz} (the tarball for GUAVA, the
+GAP error-correcting codes package) from
+\url{http://www.gap-system.org/Packages/guava.html}.
 \item
+Implement your functionality there.
+Suppose you have a directory \code{sagefiles}.
+Within that, create a subdirectory \code{guava-3.6}.
+Extract the tarball into the directory \code{guava-3.6}.
 \item
+Type \code{hg_sage.status()} and
+\code{hg_sage.diff()} to see exactly what
+you've done (you can pass options to diff to
+see information about certain files).
+Create a file \code{guava-3.6/spkg-install} with the following
+contents:
 %skip
 \begin{verbatim}
+--------------------------------------------------------
+| SAGE Version 1.6.1, Build Date: 2007-01-14           |
+| Distributed under the GNU General Public License V2. |
+--------------------------------------------------------
+#!/bin/sh
+Loading SAGE library. Current Mercurial branch is: myver
+if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.10 ]; then
+    echo "This package requires that GAP 4.4.10 is installed."
+    exit 1
+fi
+sage: hg_sage.status()
+Getting status of modified or unknown files:
+cd "/home/jaap/sage/devel/sage" && hg status
+M sage/databases/sloane_functions.py
+cp -pr guava-3.6 $SAGE_ROOT/local/lib/gap-4.4.10/pkg/
+---
+Branch: myver
+sage: hg_sage.diff()
+cd "/home/jaap/sage/devel/sage" && hg diff  | less
+cd $SAGE_ROOT/local/lib/gap-4.4.10/pkg/guava-3.6
+./configure ../..
+make
 \end{verbatim}
+\item Type \code{hg_sage.add([filenames])} to add new
+files to your repository.
+\item Type \code{hg_sage.commit([optional filenames])}
+to commit the changes in files to the repository--if no filenames are given all
+files are committed. First the output of hg diff is piped through less, look at
+it or just enter "q". Then you are asked to give your comment on the changes.
+Type "i", write some meaningful one liner, hit 'Escape' and type ":wq".
+(Note: vi is the default editor. In bash, to make emacs the default, type
+\code{export EDITOR=emacs}.)
+%skip
+\begin{verbatim}
+sage: hg_sage.commit()
+cd "/home/jaap/sage/devel/sage" && hg diff  | less
+cd "/home/jaap/sage/devel/sage" && hg commit
+\end{verbatim}
+Make it executable with \code{chmod +x spkg-install}.
 \item
+Pull your changes back into your main repository: Note the argument to
+hg_sage.pull() can be the full name starting with a '/' or the branch
+name 'myver'.
+In the directory \code{sagefiles},
+issue the command
 %skip
 \begin{verbatim}
+sage -br main
+    sage -pkg guava-3.6
+\end{verbatim}
+This will do some checks, run tar/bz2, and create the spkg file,
+\code{guava-3.6.spkg}, in the
+directory \code{sagefiles},
+\end{itemize}
+sage: hg_sage.pull('myver') #or hg_sage.pull('/home/jaap/sage/devel/sage-myver')
+cd "/home/jaap/sage/devel/sage" && hg status
+cd "/home/jaap/sage/devel/sage" && hg status
+cd "/home/jaap/sage/devel/sage" && hg pull -u /home/jaap/sage/devel/sage-myver
+pulling from /home/jaap/sage/devel/sage-myver
+searching for changes
+adding changesets
+adding manifests
+adding file changes
+added 1 changesets with 1 changes to 1 files
+files updated, 0 files merged, 0 files removed, 0 files unresolved
+If it says use 'hg merge' above, then you should
+type hg_sage.merge(), where hg_sage is the name
+of the repository you are using.  This might not
+work with the notebook yet.
+To test this, within the \code{sagefiles} directory, type
+%skip
+\begin{verbatim}
+    sage -i guava-3.6.spkg
+\end{verbatim}
+It should install and compile the guava program in sage.
+To test its functionality, one could do the following:
+sage: hg_sage.update()
+cd "/home/jaap/sage/devel/sage" && hg update
+files updated, 0 files merged, 0 files removed, 0 files unresolved
+\begin{verbatim}
+sage: gap(`LoadPackage("guava")')          # requires optional package
+true
+sage: gap(`HammingCode(3,3)')              # requires optional package
+a linear [13,10,3]1 Hamming (3,3) code over GF(3)
 \end{verbatim}
+\item
+You can make {\it all} changes in the repository you're working in
+as a bundle by typing \code{hg_sage.bundle('mybundle')} (this
+creates an hg bundle \code{mybundle.hg}).  Alternatively,
+you can export any particular changeset as plain text patches by
+typing \code{hg_sage.export(...)}; note that each individual changeset
+is recorded as a different patch.
+\code{hg_sage.export(...)} needs at least the argument
+revs -- integer or list of integers (revision numbers); use the hg_sage.log()
+function to see them. An optional second argument is a 'patch_filename',
+default is '(changeset_revision_number).patch'.
+These can be applied to any other
+repository using \code{hg_sage.import_patch('patch_filename')}.
+Besides \code{hg_sage.diff(...)}, another way to see what
+changes are contained in a patch bundle is to use the command
+\code{hg_sage.inspect('patchname.hg')}. This displays the
+``diff file'' as plain text using \code{less}.
+Note that you don't get to ``cherry pick'' what you send in a bundle.
+Everything you recorded is sent.  Hence you might want to use the
+cloning, etc. above to send only from your own sage-main repository.
+Note also that if you have a repository, e.g., sage-mynewstuff
+and it's all ready to use, you can do
+               \verb+sage -hg pull <SAGE_ROOT>/devel/sage-mynewstuff+
+from your \verb+<SAGE_ROOT>/devel/sage-main repository+, and you'll
+get all the sage-mynewstuff merged in.   NOTE: read the output!
+After that you must type \code{sage -hg update}.
+\item
+  You can also start a very nice web server that allows you to
+  navigate your repository with a web browser, or pull patches from it
+  remotely:
+To  build it again if you've already installed the package, use
 %skip
 \begin{verbatim}
+sage: hg_sage.serve()
+    sage -f guava-3.6.spkg
 \end{verbatim}
-\item
-  For the occasional contributor:
-%skip
-\begin{verbatim}
-sage: hg_sage.ci()
-sage: hg_sage.send('patchname.hg')
-\end{verbatim}
+Post the file 'patchname.hg' to the trac server.
+Note that 'ci()' is an alias for 'commit()'.
+\chapter{The \sage Trac Server: Submitting Patches and Packages}
+\label{ch:trac}
+What should you do with your Mercurial patches for \sage?  You should
+post them on the \sage trac server.
+The \sage trac server, located at
+\url{http://trac.sagemath.org/sage_trac/}, is where \sage bugs are
+listed and patched, new code is posted and reviewed, and ideas for
+extending and improving \sage are discussed.  Thus if you find a bug
+in \sage, or if you have new code to submit, or if you have
+corrections for the documentation, you should post on the trac server.
+Items on the server are called ``tickets'', and anyone may browse the
+tickets: just visit \url{http://trac.sagemath.org/sage_trac/report}.
+You need to open an account, though, if you want to comment on a
+ticket, submit a patch, or create a new ticket.  See the web page
+\url{http://wiki.sagemath.org/TracGuidelines} for more information
+about obtaining an account, as well as general guidelines for using
+the trac server.
+\section{Reporting Bugs}
+``The first step is admitting you have a problem.''
+If you think you've found a bug in \sage, you should first search
+through the Google groups \code{sage-devel} and \code{sage-support}
+for postings related to your possible bug: maybe it has already been
+discussed.  You should also search the trac server to see if anyone
+else has opened a ticket about your bug.
+If you don't find anything, and you're not sure that you've found a
+bug, ask about it on \code{sage-devel}.
+If you don't find anything, and if you're positive you've found a bug,
+open a new ticket on the trac server.  As mentioned above, you need an
+account to do this.  To report a bug: log in and click on the ``New
+ticket'' button.  Type a meaningful one-liner in the ``Short summary''
+box, with more information in the larger box below.  You should
+include an explicit, reproducible example illustrating your bug
+(and/or the steps required to reproduce the buggy behavior).  You
+should also include the version of \sage (and any relevant packages)
+you are using, and operating system information, being precise as
+possible (32-bit, 64-bit, ...).
+Between the ``Summary'' and ``Full description'' boxes, there is a
+place to choose the ``Type'' of the ticket: ``Defect'',
+``Enhancement'', or ``Task''.  Use your best judgment here; a bug
+should probably be reported as a ``Defect''.
+Choose a priority for your bug, keeping in mind that the ``blocker''
+label should be used very sparingly.  Also pick a component for your
+bug; this is sometimes straightforward -- if your bug deals with
+\sage's calculus implementation, choose ``calculus''.  If it is not
+obvious, do your best.  Choose a milestone; if you're not sure what to
+choose, just choose the numbered version of sage from the menu
+(``sage-3.1.4'', for example).  Type in some helpful keywords.  In the
+box labeled ``Assign to'', type ``somebody'' if you're not sure what
+else to do.
+Hit the ``Preview'' button to make sure everything looks okay, and
+then hit ``Submit ticket''.
+\section{Guidelines for Opening Tickets}
+In addition to bug reports, you should also open a ticket if you have
+some new code which extends \sage's capabilities.  If you have a
+feature request, start a discussion on \code{sage-devel} first, and
+then if there seems to be general agreement that you have a good idea,
+open a ticket describing the idea.
+Other comments:
+\begin{itemize}
+\item Before opening a ticket, make sure that nobody else has opened a
+ticket about the same or closely related issue.
+\item It is much better to open several specific tickets than one that
+is very broad.  Indeed, a single ticket which deals with lots of
+different issues can be quite problematic, and should be avoided.
+\item Be precise: If foo doesn't work on OS X but is fine on Linux,
+mention that in the title; also use the keyword option so that
+searches will pick up the issue.
+\item The problem described in the ticket must be solvable.  For
+example, it would be silly to open a ticket whose purpose was ``Make
+\sage the best mathematical software in the world''. There is no metric
+to measure this properly and it is highly subjective.
 \end{itemize}
-You can browse the official repositories for \SAGE with your web browser here:
-    \verb+http://sagemath.org/sage/hg+
-These are served using a cgi-bin script.
-\section{Updating To the Latest Official \SAGE Library Source Code}
-If you're wary of messing things up, before typing the commands
-explained below, first type from the command line
-\code{sage -clone test}.  To switch back to your main repository (the default),
-type \code{sage -b main}.
+To update to the latest \SAGE Mercurial source
+code type \code{hg_sage.pull()}.  You may have to type
+\code{hg_sage.update()}.
+\section{Patching Bugs/Working on Tickets}
+At this stage, you might get a message like:
+\begin{verbatim}
+remote changed .hgtags which local deleted
+(k)eep or (d)elete?
+\end{verbatim}
+Type \verb+k <RETURN>+. You might then get
+\begin{verbatim}
+remote changed export which local deleted
+(k)eep or (d)elete?
+\end{verbatim}
+Type \verb+k <RETURN>+.
+If you have code which fixes a bug or deals with some issue in \sage,
+here's what to do: first, use Mercurial to create a patch file.  If
+the issue has been reported as a ticket on the trac server, attach your
+patch file to that ticket: go to the ticket, click on the ``Attach
+File'' button, and follow the directions.  On the ticket page, you
+should add a comment explaining your patch, and you should also
+change the summary for the ticket from \texttt{description of bug} to
+\texttt{[with patch, needs review] description of bug}.
+You might then get
+\verb+merging sage/...+
+Now kdiff3 opens with some text highlighted
+in yellow and red. The bar at the bottom says "Number of remaining
+unsolved conflicts: ...".
+The conflict appears to be in the file named in the lower part
+of the kdiff3 window. Clicked on "merge" on the
+top bar, pick one of the options in the pop-down  menu,
+then click on "save" and quit.
+If there is no trac ticket associated to this issue, create one (as
+explained in the previous sections) describing the issue and your
+solution, attach your patch, and give it a summary of the form
+\texttt{[with patch, needs review] description of bug goes here}.
-If you don't have kdiff3, install it with \code{apt get kdiff3} or
-\code{yum install kdiff3}.
-If it says you need to, type
-\code{hg_sage.commit()}
-to finish updating your mercurial repository.
+\section{Reviewing Patches}
+\chapter{Miscellaneous}
+\section{Circular Imports}
+This needs to be cleaned up, but an irc log about this common problem
+is better than nothing:
+\begin{verbatim}
+ncalexan1: from sage.schemes.plane_curves.curve import Curve_generic
+[5:08pm] ncalexan1: gives an ImportError
+was389: You can import anything from a file *unless* you set yourself up to
+have a *circular* import loop.
+[5:11pm] was389: The interpreter actually runs through the code of the file,
+so it's not possible to circularly import usig
+[5:11pm] ncalexan1: That's possible.
+[5:11pm] was389: from blah import foo
+[5:11pm] ncalexan1: Oh.
+[5:11pm] was389: You might have to either (1) do import
+sage.schemes.plane_curves.curve and explicitly access
+[5:12pm] was389: Curve_generic, or (2) wait to import Curve_generic until
+where it is used.
+\end{verbatim}
+All code that goes into \sage is peer-reviewed, to ensure that the
+conventions discussed in this manual are followed, to make sure that
+there are sufficient examples and doctests in the documentation, and
+to try to make sure that the code does, mathematically, what it is
+supposed to.
+\section{Weird Issues}
+\begin{enumerate}
+\item
 If you see
+\begin{verbatim}
+> > - "*** glibc detected *** free(): invalid pointer: 0x0846a684 ***"
+\end{verbatim}
 when leaving \sage, you might wonder why.
+If someone (other than you) has posted a patch for a ticket on the
+trac server, you can review it.  Look at the patch (by clicking on the
+file name in the list of attachments) to see if it makes sense.
+Download it (from the window displaying the patch, see the
+``Download'' option at the bottom of the page).  Apply it (using
+\code{hg_sage.patch('filename')}, for example) to your copy of \sage,
+and build \sage with the new code by typing \code{sage -b} (for
+patches to .py files) or \code{sage -ba} (for patches to .pyx files).
+Martin Albrecht finally figured out how to solve this strange bug. The
+scenario: If you write a PYREX wrapper for some C code which needs to
+be linked to the GMP library and you load the extension you wrote
+during runtime into \sage you'll get such error messages on exit or
+even more strange behavior.
+This vanishes completely if you load your extension module during the
+startup of \SAGE like the other library wrappers/PYREX code (e.g. in
+\code{sage/libs/all.py}). This is because when you load the module the
+GMP library gets somehow reinitialized and all your prior GMP
+variables are not valid anymore. So when they are freed/cleared on
+exit you get this error stated above. Some objects relying on GMP are
+created during startup of \SAGE so you'll see this behavior as well if
+the first thing you do on the prompt is to load your library.
+Now ask yourself questions like these:
+\begin{itemize}
+\item Does the new source code make sense?
+\item When you run it in \sage, does it fix the problem reported on
+the ticket?
+\item Does it fail to introduce any new problems?
+\item Is it documented sufficiently, including both explanations and
+doctests?  (This is \textbf{very} important: all code in \sage must
+have doctests, so even if the patch is for code which didn't have a
+doctest before, the new version must include one.)
+\item In particular, is there a doctest illustrating that the bug has
+been fixed?  If a function used to give the wrong answer and this
+patch fixes that, then if possible, it should include a doctest
+illustrating its new success.
+\end{itemize}
+\end{enumerate}
+If the answers to these and other such reasonable questions are yes,
+then you might want to give the patch a positive review.  On the main
+ticket page, write a comment in the box and change the summary from
+\texttt{[with patch, needs review] description of bug} to
+\texttt{[with patch, positive review] description of bug}.  If you
+feel there are issues with the patch, explain them in the comment box,
+and change the summary to \texttt{[with patch, negative review]
+description of bug}, or \texttt{[with patch, needs work]
+description of bug}, or \texttt{[with patch, positive review pending fixes]
+description of bug}, or something similar.  Browse the tickets on the
+trac server to see how things are done.
+\section{Benchmarking}
+\label{sec:Benchmarking}
+By the way, if you review a patch which deals with the \sage manuals,
+say, instead of the source code, then you need to use
+\code{hg_doc.patch('filename')} instead of
+\code{hg_sage.patch('filename')} to apply it, and you need to follow
+the directions in Chapter~\ref{ch:manuals} to build the new
+documentation.
-\begin{verbatim}
+http://sage.math.washington.edu:8100/william_32bit_add?edit
+\section{Closing Tickets}
+{{{
+if is_64_bit:
+    word = 64
+else:
+    word = 32
+}}}
+Don't close tickets.  That is the job of the \sage administrators.  If
+you feel strongly that a ticket should be closed or deleted, send
+email to the current release manager explaining the situation.
-{{{
-def testpy(n, op):
-    a = long(ZZ.random(2^(word*n)))
-    b = long(ZZ.random(2^(word*n)))
-    N = 1000000
-    t = cputime()
-    for i in range(N):
-        c = op(a, b)
-    return cputime(t)
-}}}
-{{{
-def testgmp(n, op):
-    a = ZZ.random(2^(word*n))
-    b = ZZ.random(2^(word*n))
-    N = 1000000
-    t = cputime()
-    for i in range(N):
-        c = op(a, b)
-    return cputime(t)
-}}}
-{{{
-op=operator.add
-for n in range(1,20):
-    print 4*n, testpy(4*n,op) / testgmp(4*n,op)
-///
-0.336633663366
-0.378640776699
-0.388349514563
-0.450980392157
-0.509803921569
-0.509259259259
-0.527272727273
-0.504587155963
-0.653846153846
-0.590476190476
-0.681818181818
-0.718181818182
-0.654867256637
-0.648648648649
-0.973451327434
-0.956896551724
-0.920353982301
-1.07894736842
-1.04347826087
-}}}
-{{{
-op=operator.mul
-for n in range(1,10):
-    print n, testpy(n,op) / testgmp(n,op)
-///
-0.475
-0.5
-0.527777777778
-0.663716814159
-0.801724137931
-0.991596638655
-1.21551724138
-1.41176470588
-1.56589147287
-}}}
 \end{verbatim}
+\appendix
+\begin{verbatim}
+http://sage.math.washington.edu:8100/william_64bit_add
+{{{
+if is_64_bit:
+    word = 64
+else:
+    word = 32
+print word
+///
+}}}
+{{{
+def testpy(n, op):
+    a = long(ZZ.random(2^(word*n)))
+    b = long(ZZ.random(2^(word*n)))
+    N = 1000000
+    t = cputime()
+    for i in range(N):
+        c = op(a, b)
+    return cputime(t)
+}}}
+{{{
+def testgmp(n, op):
+    a = ZZ.random(2^(word*n))
+    b = ZZ.random(2^(word*n))
+    N = 1000000
+    t = cputime()
+    for i in range(N):
+        c = op(a, b)
+    return cputime(t)
+}}}
+{{{
+op=operator.add
+for n in range(1,20):
+    print 4*n, testpy(4*n,op) / testgmp(4*n,op)
+///
+0.478991596639
+0.5
+0.526717557252
+0.55905511811
+0.5859375
+0.615384615385
+0.709923664122
+0.731884057971
+0.833333333333
+0.873134328358
+0.878571428571
+0.85401459854
+0.950704225352
+1.1
+1.12121212121
+0.977941176471
+}}}
+{{{
+op=operator.mul
+for n in range(1,10):
+    print n, testpy(n,op) / testgmp(n,op)
+///
+0.487179487179
+0.747826086957
+1.13913043478
+1.56896551724
+2.0859375
+2.63846153846
+3.35384615385
+4.38636363636
+5.0962962963
+}}}
+\end{verbatim}
+\chapter{History and License}
+\chapter{\sage's License}
 \input{license}

Download in other formats:

Original Format