Ticket #3905: prog.patch

File prog.patch, 160.2 KB (added by jhpalmieri, 11 years ago)
  • commontex/macros-new.tex

    # HG changeset patch
    # User J. H. Palmieri <palmieri@math.washington.edu>
    # Date 1219199147 25200
    # Node ID 8f9275ea4358f44711241bbc022128ef6a46d202
    # Parent  bba79837cb391923027900edb059116505472ab7
    revising programming guide
    
    diff -r bba79837cb39 -r 8f9275ea4358 commontex/macros-new.tex
    a b  
    4545% when run through latex2html.  See the command \newbf above for an
    4646% application of this.  See the latex2html docs for more information.
    4747\usepackage{html}
     48
     49\usepackage{verbatim}
     50% the following is used for nesting verbatim environments, used in prog/prog.tex
     51\newenvironment{newverbatim}%
     52  \verbatim%
     53  {\endverbatim}
    4854
    4955\newcommand{\by}[1]{\par\noindent{\bf AUTHOR:} #1\par}
    5056
  • prog/prog.tex

    diff -r bba79837cb39 -r 8f9275ea4358 prog/prog.tex
    a b  
     1% see 3393, 3882
     2
     3% coercion section
     4% mutability section should be rewritten, expanded.
     5% benchmarking?
     6
    17\documentclass{manual}
    28\usepackage{textcomp}
    3 \include{macros}
     9
     10\include{macros-new}
    411% a hack to get around another hack.
    512\ifpdf
    613\usepackage{url}
    714\else
    815\fi
    916
    10 
    1117%\newcommand{\todo}[1]{\footnote{1}}
    1218\newcommand{\todo}[1]{}
    1319
    1420\newcommand{\swlist}{GAP, GSL, Matplotlib, Maxima, MWRANK, NetworkX,
    15   NTL, Numpy, PARI and Singular}
     21  NTL, Numpy, PARI and Singular\xspace}
    1622
    1723\usepackage{color}
    1824\definecolor{hue}{rgb}{.202, .602, .58}
     
    3137\begin{document}
    3238
    3339
    34 \title{\SAGE Programming Guide}
     40\title{\SAGE Developer's Guide}
    3541
    36 \author{William Stein%\thanks{Associate Professor, University of Washington, wstein@math.washington.edu.}
    37 \ \ and David Joyner%\thanks{Mathematics Department, US Naval Academy, wdj@usna.edu}
    38 }
     42\author{William Stein and David Joyner}
    3943
    4044\maketitle
    4145
     
    4852\begin{abstract}
    4953
    5054\noindent
    51 Absolutely {\bf everybody} who uses \SAGE should contribute something
     55\SAGE is a free mathematics software system. It is implemented using
     56Python, Cython, and C++, and uses \swlist.  It is free and open
     57source, and is available under the terms of the GNU Public License.
     58
     59Everybody who uses \SAGE should contribute something
    5260back to \SAGE at some point.  Implement a new function, add examples
    5361to the documentation, find bugs and typos, fix a bug, create a new
    5462class, create a fast new C library, etc.  This book is a guide about
    55 how to contribute code back to \SAGE.
     63how to contribute to \SAGE.
    5664
    57 This document describes how to write programs using \SAGE, and also
    58 how to modify and extend the core \SAGE libraries.
    59 
    60 \SAGE stands for Software for Algebra and Geometry Experimentation.
    61 It is implemented using Python, Cython, and C++, and uses \swlist.
    62 It is free and open source, and is available under the terms of the
    63 GNU Public License.
     65This document describes how to write programs using \SAGE,
     66how to modify and extend the core \SAGE libraries, and how to modify
     67\sage's documentation.  It also discusses how to share your new and
     68modified code with other \SAGE users.
    6469
    6570\end{abstract}
    6671
    6772
    6873\tableofcontents
    6974
    70 \todo{Remove I and me}
     75
     76\part{Writing Code for \sage}
    7177
    7278\chapter{Introduction}
    7379If there is something you would like to implement and make
     
    9399contains sophisticated, optimized number theory algorithms.  Notably
    94100absent from this triad is a good system for exact linear algebra
    95101(something MAGMA does extremely well), but this gap is being filled by
    96 code being written for \SAGE. \todo{Say something about GSL,
    97   Matplotlib, Maxima, MWRANK, NetworkX, NTL, Numpy}
    98 %Unfortunately, Singular, GAP, and PARI
    99 %have vastly different languages and conventions.
     102code written for \SAGE.
     103\todo{Say something about GSL, Matplotlib, Maxima, MWRANK, NetworkX,
     104NTL, Numpy}
    100105
    101 \SAGE is not just about gathering together functionality (something
    102 UNIX already does well), it is about providing a clear, systematic and
     106\SAGE is not just about gathering together functionality, it is about
     107providing a clear, systematic and
    103108consistent way to access a large number of algorithms, in a coherent
    104109framework that makes sense mathematically.  In the design of \SAGE, the
    105110semantics of objects, the definitions, etc., are informed by how the
     
    108113This document was authored by William Stein, David Joyner and others
    109114with the editorial help of Iftikhar Burhanuddin.
    110115
    111 \chapter{\SAGE{} Coding Conventions}\label{ch:conventions}
    112 To meet the goal of making \sage easy to read, all Python/Cython code
    113 that is included with \sage should adhere to the style conventions
    114 mentioned in this chapter.
    115116
    116 \section{Coding Conventions}
    117 Follow the standard Python formatting rules when writing code for Sage,
    118 as explained at \url{http://www.python.org/doc/essays/styleguide.html}.
     117
     118\chapter{Conventions for Coding in \SAGE}\label{ch:conventions}
     119
     120To meet the goal of making \sage easy to read, maintain, and improve,
     121all Python/Cython code that is included with \sage should adhere to
     122the style conventions discussed in this chapter.
     123
     124\section{Python Coding Conventions}
     125
     126Follow the standard Python formatting rules when writing code for \sage,
     127as explained at
     128\url{http://www.python.org/dev/peps/pep-0008/} and
     129\url{http://www.python.org/dev/peps/pep-0257/}.
    119130In particular,
    120131\begin{itemize}
    121132\item Use 4 spaces for indentation levels.  Do not use tabs as they
     
    128139\begin{verbatim}
    129140        def set_some_value()
    130141\end{verbatim}
    131      instead of:
     142     instead of CamelCase:
    132143\begin{verbatim}
    133144        def SetSomeValue()
    134145\end{verbatim}
     
    137148\end{itemize}
    138149
    139150
    140 \section{File and Directory names}
     151\section{File and Directory Names}
     152
    141153Python \sage library code uses the following conventions.  Directory
    142154names may be plural (e.g., \code{rings}) and file names are almost
    143155always singular (e.g., \code{polynomial\_ring.py}).  Note that the
    144156file \code{polynomial\_ring.py} might still contain definitions of
    145157several different types of polynomial rings.
    146158
    147 \note{You are strongly encouraged
     159\note{You are encouraged
    148160to include miscellaneous notes, emails, design
    149161discussions, etc., in your package.  Make these
    150162plain text files (with extension \code{.txt})
    151 in a subdirectory directory called \code{notes}.
     163in a subdirectory called \code{notes}.  (For example, see
     164\code{SAGE_ROOT/devel/sage/sage/ext/notes/}.)
    152165}
    153166
    154 %\section{Ordering of function names in a class} 
    155 %Function names
    156 %should be defined in alphabetical order, with \code{__}'s first,
    157 %then \code{_}'s, then public functions.   This is for easy of readability
    158 %and finding code.  It also encourages functions with related behavior
    159 %to be named in a similar way, which makes it easier to find them
    160 %in the documentation and when using tab completion from the interpreter.
    161 
    162167
    163168\section{Headings of \SAGE Library Code Files}
    164 The top of each \sage code file should appear as follows\todo{What is \section{Tutorial} below}:
     169
     170The top of each \sage code file should follows this format:
    165171
    166172\begin{verbatim}
    167173r"""
     
    171177...
    172178
    173179AUTHORS:
    174     - YOUR NAME (2005-01-03): initial version
    175     - person (date in ISO year-month-day format): short desc
     180    -- YOUR NAME (2005-01-03): initial version
     181    -- person (date in ISO year-month-day format): short desc
    176182     ...
    177     - person (date in ISO year-month-day format): short desc
     183    -- person (date in ISO year-month-day format): short desc
    178184
    179 \section{Tutorial}
    180185 ...
    181186
    182187Lots and lots of examples.
    183188"""
    184189
    185190#*****************************************************************************
    186 #       Copyright (C) 2006 William Stein <wstein@gmail.com>
    187 #                     2006 YOUR NAME <your email>
     191#       Copyright (C) 2008 YOUR NAME <your email>
    188192#
    189193#  Distributed under the terms of the GNU General Public License (GPL)
    190194#                  http://www.gnu.org/licenses/
    191195#*****************************************************************************
    192196\end{verbatim}
    193197
    194 The following is top of the ssmod.py file, which contains the
    195 SupersingularModule code and is located at
    196 \code{SAGE_ROOT/devel/sage/sage/modular/ssmmod/ssmod.py}.
     198The following is the top of the file
     199\code{SAGE_ROOT/devel/sage/sage/modular/ssmmod/ssmod.py}, which
     200contains the Supersingular Module code.
    197201
    198202{\small
    199203
    200204\begin{verbatim}
    201 
    202205"""
    203206Module of Supersingular Points
    204207
     
    216219    [
    217220    (Vector space of degree 33 and dimension 1 over Finite Field of size 97
    218221    Basis matrix:
    219     [ 0  0  0  1 96 96  1  0 95  1  1  1  1 95  2 96  0  0 96 96  0  0 96
    220       2 96 96  2  0  0  1  1 95  0], 1),
     222    [ 0  0  0  1 96 96  1 96 96  0  2 96 96  0  1  0  1  2 95  0  1  1  0  1  0 95  0 96 95  1 96  0  2], True),
    221223    (Vector space of degree 33 and dimension 1 over Finite Field of size 97
    222224    Basis matrix:
    223     [ 0  1 96 16 75 22 81  0  0 17 17 80 80  0  0 16  1 40 74 23 57 96 81 
    224       0 23 74  0  0  0 24 73  0  0], 1),
     225    [ 0  1 96 75 16 81 22 17 17  0  0 80 80  1 16 40 74  0  0 96 81 23 57 74  0  0  0 24  0 23 73  0  0], True),
    225226    (Vector space of degree 33 and dimension 1 over Finite Field of size 97
    226227    Basis matrix:
    227     [ 0  1 96 90 90  7  7  0  0  6 91  6 91  0  0  7 13  0 91  6  0 84 90 
    228       0 91  6  0  0  0 90  7  0  0], 1)
     228    [ 0  1 96 90 90  7  7  6 91  0  0 91  6 13  7  0 91  0  0 84 90  6  0  6  0  0  0 90  0 91  7  0  0], True)
    229229    ]
    230230    sage: len(D)
    231231    9
    232232
    233233We compute a Hecker operator on a space of huge dimension!
    234234    sage: X = SupersingularModule(next_prime(100000))
    235     sage: t = X.T(2).matrix()  # long time (but still less than a minute!)
    236     sage: t.nrows()            # long time
     235    sage: t = X.T(2).matrix()            # long time (but still less than a minute!)
     236    sage: t.nrows()                      # long time
    237237    8334
     238
     239TESTS:
     240    sage: X = SupersingularModule(389)
     241    sage: T = X.T(2).matrix().change_ring(QQ)
     242    sage: d = T.decomposition()
     243    sage: len(d)
     244    6
     245    sage: [a[0].dimension() for a in d]
     246    [1, 1, 2, 3, 6, 20]
     247    sage: loads(dumps(X)) == X
     248    True
     249    sage: loads(dumps(d)) == d
     250    True
    238251"""
    239252
    240253#*****************************************************************************
     
    248261}%small
    249262
    250263All code included with \sage must be licensed under the GPL or a less
    251 restrictive license (e.g., the BSD license).  You should share
    252 copyright with William Stein by including your name at the top.  It is {\em very
     264restrictive license (e.g., the BSD license). 
     265It is {\em very
    253266  important} that you include your name in the AUTHOR log, since
    254267everybody who submits code to \sage to receive proper credit.  (If
    255268ever you feel you are not receiving proper credit for anything you
    256 submit to \sage, please let William Stein know!)
     269submit to \sage, please let the development team know!)
    257270
    258271\section{Documentation Strings}\label{sec:conv-docs}
    259 {\bf Every} function should have a docstring that includes
     272
     273\textbf{Every} function must have a docstring that includes
    260274the following information:
    261275\begin{itemize}
    262276\item A one-sentence description of the function, followed by a blank line.
     
    268282\item An EXAMPLES block for examples. This is {\em not} optional.
    269283\item A NOTES block for special notes (optional). Include information
    270284  such as algorithm used, references, etc.
    271 \item An AUTHORS block (important!).
     285\item An AUTHORS block (optional, but encouraged for important
     286functions, so users can see from the docstring who wrote it and
     287therefore whom to contact if they have questions).
    272288\end{itemize}
    273289Use the following template when documenting functions. Note the
    274290indentation:
    275291%skip
    276292\begin{verbatim}
    277293def point(self, x=1, y=2):
    278     r"""           # note the r for "raw string"
     294    r"""
    279295    This function returns the point $(x^5,y)$.
    280296
    281297    INPUT:
     
    321337\begin{itemize}
    322338
    323339\item Use nice \Latex formating everywhere but in the
    324   \code{INPUT/OUTPUT} blocks.
     340  \code{INPUT/OUTPUT} blocks.  If you use backslashes, either use
     341  double backslashes or place an {\tt r} right before the first triple
     342  opening quote. For example,
     343%skip
     344\begin{verbatim}
     345def cos(x):
     346    """
     347    Returns $\\cos(x)$.
     348    """
     349
     350def sin(x):
     351    r"""
     352    Returns $\sin(x)$.
     353    """
     354\end{verbatim}
     355
     356Also, the non-standard \Latex macros that are used when typesetting the
     357documentation are defined in
     358\code{SAGE_ROOT/doc/commontex/macros.tex}.  If you need to add more
     359macros, add them there and post a patch file on the \sage trac server
     360(Chapter~\ref{ch:trac}) so that they can be included in the next
     361version of \sage.
    325362
    326363\item Liberally describe what the examples do.  Note that there must
    327364  be a blank line after the example code and before the explanatory
     
    334371  regular basis, and are crucial for the quality and adaptability of
    335372  \sage.  Without such examples, small changes to one part of \sage
    336373  that break something else might not go seen until much later when
    337   someone uses the system, which is frustrating.
     374  someone uses the system, which is unacceptable.
    338375\end{itemize}
    339376 
    340377The code in the examples should pass automatic testing.  This means
    341378that if the above code is in the file \code{f.py} (or \code{f.sage}),
    342379then \code{sage -t f.py} should not give any error messages.  Testing
    343380occurs with full \sage preparsing of input within the standard \sage
    344 shell environment.  {\em Important.}\todo{Clarify with an
    345   example/pathology and explanation. } The file \code{f.py} is not
     381shell environment, as described in Section~\ref{sec:Preparsing}.
     382{\em Important:} The file \code{f.py} is not
    346383imported when running tests unless you have arranged that it be
    347384imported into your \sage environment, i.e., unless its functions are
    348 available when you start \sage using the \code{sage} command.
     385available when you start \sage using the \code{sage} command.  For
     386example, the function \code{cdd_convert} in the file
     387\code{SAGE_ROOT/devel/sage/sage/geometry/polyhedra.py} includes
     388an EXAMPLES block containing the following:
     389\begin{verbatim}
     390    sage: from sage.geometry.polyhedra import cdd_convert
     391    sage: cdd_convert(' 1 1 0 0')
     392    [1, 1, 0, 0]
     393\end{verbatim}
     394\sage doesn't know about the function \code{cdd_convert} by default,
     395so it needs to be imported before it is tested; hence the first line
     396in the example.
    349397
    350 \subsection{Further conventions for automated testing of examples}
     398
     399
     400\subsection{Further Conventions for Automated Testing of Examples}
    351401\label{sec:Further-conventions}
    352402The Python script \code{local/bin/sage-doctest} implements
    353 documentation testing in \sage (see Chapter~\ref{ch:testing} for more
     403documentation testing in \sage (see Section~\ref{ch:testing} for more
    354404details).  When writing documentation, keep the following points in
    355405mind:
    356406\begin{itemize}
     
    359409  \code{2/3} as a rational instead of the Python int \code{0}. For
    360410  more information on preparsing see Section \ref{sec:Preparsing}.
    361411
    362 \item If a test line contains the text \code{random} it is executed by
     412\item If a test line contains the text \code{random}, it is executed by
    363413  \code{sage-doctest} but \code{sage-doctest} does not check that the
    364414  output agrees with the output in the documentation string.
     415  For example, the docstring for the \code{__hash__} method for
     416  \code{CombinatorialObject} in
     417  \code{SAGE_ROOT/devel/sage/sage/combinat/combinat.py} includes the
     418  lines
     419%skip
     420\begin{verbatim}
     421    sage: hash(c) #random
     422    1335416675971793195
     423    sage: c._hash #random
     424    1335416675971793195
     425\end{verbatim}
    365426
    366427\item If a line contains the text \code{long time} then that line is
    367428  not tested unless the {\code -long} option is given, e.g.,
     
    370431  \sage development, but will get run before major releases.  No
    371432  example should take more than about 30 seconds.
    372433
    373   Here is the \code{Lseries} function docstring which contains the phrase
    374   \code{long time} and is taken from
    375 \code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_rational.py}
    376 
     434  For example, here is part of the docstring from the \code{regulator} method for
     435  rational elliptic curves, from the file
     436  \code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_rational.py}:
     437%skip
    377438\begin{verbatim}
    378     def Lseries(self, s):
    379         r"""
    380         Returns the value of the L-series of the elliptic curve E at s, where s
    381         must be a real number.
    382 
    383         Use self.Lseries_extended for s complex.
    384 
    385         \note{If the conductor of the curve is large, say $>10^{12}$,
    386         then this function will take a very long time, since it uses
    387         an $O(\sqrt{N})$ algorithm.}
    388 
    389         EXAMPLES:
    390             sage: E = EllipticCurve([1,2,3,4,5])
    391             sage: E.Lseries(1)
    392             0.000000000000000
    393             sage: E.Lseries('1.1')       # long time (!)
    394             0.28549100767814833
     439            sage: E = EllipticCurve([0, 0, 1, -1, 0])
     440            sage: E.regulator()              # long time (1 second)
     441            0.0511114082399688
    395442\end{verbatim}
    396 
    397443
    398444\item If a line contains \code{todo: not implemented}, it is never
    399445tested.   It is good to include lines like this to make clear
     
    409455\code{sage -t -optional f.py}.  (Note that \code{-optional}
    410456must not be the first argument to \code{sage}.)  Use this
    411457to include doctests that require optional packages.
    412 For example,
     458For example, the docstring for \code{_magma_init_} in the
     459class \code{EllipticCurve_finite_field} from the
     460\code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_finite_field.py}
     461contains
    413462\begin{verbatim}
    414     sage: E.padic_regulator(5)  # requires optional MAGMA package
    415     xxx
     463    sage: EllipticCurve(GF(41),[2,5])._magma_init_() # optional -- requires Magma
     464    'EllipticCurve([_sage_[1]|GF(41)!0,GF(41)!0,GF(41)!0,GF(41)!2,GF(41)!5])'
    416465\end{verbatim}
    417466
    418467\item If the entire documentation string contains all three words
     
    424473
    425474\end{itemize}
    426475
    427 Using \code{sage_search} from the \sage prompt or \code{grep} one can
    428 easily find aforementioned keywords and in the case of \code{todo: not
    429   implemented} use the results to motivate further development on \sage.
     476Using \code{search_src} from the \sage prompt (or \code{grep}), one can
     477easily find the aforementioned keywords.  In the case of \code{todo: not
     478 implemented}, one can use the results of such a search to direct
     479further development on \sage.
    430480
    431 \section{Exceptions}
    432 Whenever code such as the following should be avoided:
     481
     482\section{Automated Testing}\label{ch:testing}
     483
     484This section describes \sage's automated testing of test files of the
     485following types: \code{.py .pyx .sage .tex}.  Briefly, use \code{sage -t
     486  <file>} to test that the examples in \code{<file>} behave exactly as
     487claimed.  See the following subsections for more details.
     488See also Section~\ref{sec:conv-docs} for a discussion of how to
     489include examples in documentation strings and what conventions to
     490follow.
     491
     492\subsection{Testing .py, .pyx and .sage Files}\label{sec:test:python}
     493
     494Run \code{sage -t <filename.py>} to test that all code examples in
     495\code{filename.py}.  Similar remarks apply to \code{.sage} and
     496\code{.pyx} files.
     497
     498\begin{verbatim}
     499  sage -t [--verbose] [--optional]  [files and directories ... ]
     500\end{verbatim}
     501
     502When you run \code{sage -t <filename.py>}, \sage makes a copy of
     503\code{<filename.py>} with all the \code{sage} prompts replaced by
     504\code{{>}{>}>}, then uses the standard Python doctest framework to test
     505the documentation.  More precisely,
     506the Python script \code{local/bin/sage-doctest} implements
     507documentation testing.  It does the following when asked
     508to test a file \code{foo.py} or \code{foo.sage}.
     509\begin{enumerate}
     510
     511\item Creates the directory \code{.doctest} if it doesn't
     512exist and the file \code{.doctest/foo.py}.
     513
     514\item The file \code{.doctest_foo.py} contains functions for
     515each docstring in \code{foo.py}, but with all \sage preparsing
     516applied and with \code{from sage.all import *} at the top.
     517The function documentation is thus standard Python with
     518\code{{>}{>}>} prompts.
     519
     520\item \code{sage-doctest} then runs \sage's Python interpreter
     521on \code{.doctest_foo.py}. 
     522\end{enumerate}
     523
     524\note{{\em Note that line numbers in the errors 
     525you might see apply to the file \code{.doctest_foo.py}, not
     526to the original file \code{foo.py}!}  If you get errors,
     527feel free to inspect \code{.doctest_foo.py}; it is never
     528automatically deleted by \sage.}
     529
     530Your file passes these tests if the code in it will run when entered
     531at the \code{sage:} prompt with no special imports.  Thus users are
     532guaranteed to be able to exactly copy code out of the examples you
     533write for the documentation and have them work.
     534
     535
     536
     537\subsection{Testing \Latex Documentation}
     538
     539Run \code{sage -t <filename.tex>} to test the examples in
     540verbatim environments in \Latex documentation.   Put
     541\code{\%skip}
     542on the line right before a verbatim environment to have that
     543example skipped when testing the file.   
     544
     545\sage creates a file \code{.doctest_filename.py} and tests
     546it just as for \code{.py}, \code{.pyx} and \code{.sage}
     547files.   In order to skip testing of a block of code
     548in a verbatim environment, put the \Latex comment
     549\code{\%skip} on the previous line.
     550
     551Of course in \Latex files, one
     552often inserts explanatory texts between different
     553verbatim environments.    To link together verbatim
     554environments use the \code{\%link} comment.  For
     555example:
    433556%skip
    434 \begin{verbatim}
    435 try:
    436     some code
    437 except:               # bad
    438     more code
    439 \end{verbatim}
    440 Instead catch specific exceptions. For example,
    441 \begin{verbatim}
    442 try:
    443     return self.__coordinate_ring
    444 except (AttributeError, other exceptions), msg:           # Good
    445     more code to compute something
    446 \end{verbatim}
    447 Note that the syntax in the except is to list all the exceptions
    448 that are caught as a tuple, followed by the variable that holds
    449 the message.
     557\begin{newverbatim}
     558    \begin{verbatim}
     559    sage: a = 1
     560    \end{verbatim}%link
    450561
    451 If you don't have any exceptions explicitly listed (as a tuple), your
    452 code will catch absolutely anything, including ctrl-C and alarms,
    453 and this will lead to confusion.
     562    Next we add 1 to \code{a}.
     563    %link
     564    \begin{verbatim}
     565    sage: 1 + a
     566    2
     567    \end{verbatim}
     568\end{newverbatim}
    454569
    455 \section{Optional Packages} If an optional package is required for a
    456 certain function to work, that function should fail gracefully ---
    457 perhaps using a \code{try} and \code{except} block --- when the optional
    458 package is not available, and should give a hint about how to install
    459 it.  For example, typing \code{sage -optional} gives a list of all
    460 optional packages, so suggest to the user that they type that.
     570See \code{SAGE_ROOT/doc/tut/tut.tex} for many examples
     571of how to include automated testing in \Latex documentation
     572for \sage.
     573
     574
     575
     576\section{Randomized Testing}
     577\label{ch:randomtesting}
     578
     579In addition to all the examples in your docstrings, which serve as
     580both demonstrations and tests of your code, you should consider creating a
     581test suite.  Think of this as a program that will run for a while and
     582``tries'' to crash your code using randomly generated input.  Your
     583test code should define a class \class{Test} with a {\tt random()}
     584method that runs random tests.  These are all assembled together
     585later, and each test is run for a certain amount of time on a regular
     586basis.
     587
     588For example, see the file \code{SAGE_ROOT/devel/sage/sage/modular/modsym/tests.py}.
     589
     590
     591
     592\chapter{Coding in Python for \SAGE}
     593
     594This chapter discusses some issues with, and advice for, coding in \SAGE.
     595
     596
     597\section{Design}
     598
     599If you are planning to develop some new code for \sage, design is
     600important.  So think about what your program will do and how that fits
     601into the structure of \SAGE.  In particular, much of \SAGE is
     602implemented in the object-oriented language Python, and there is a
     603hierarchy of classes that organize code and functionality.  For
     604example, if you implement elements of a ring your class should derive
     605from \class{sage.structure.element.RingElement}, rather than starting
     606from scratch.  Try to figure out how your code should fit in with
     607other \sage code, and design it accordingly.
     608
    461609
    462610\section{Special \sage Functions}
    463611
     
    472620also has special methods.  They are typically of the form
    473621\code{_XXX_}.  (In a few cases the trailing underscore is not
    474622included, but this will be changed so that the trailing underscore is
    475 always included.)  This section describes all special methods.
     623always included.)  This section describes these special methods.
    476624
    477625All objects in \sage should derive from the Cython extension
    478626class \code{SageObject}:
     
    483631class MyClass(SageObject,...):
    484632    ...
    485633\end{verbatim}
     634or from some other already existing \sage class:
     635%skip
     636\begin{verbatim}
     637from sage.rings.ring import Algebra
     638
     639class MyFavoriteAlgebra(Algebra):
     640    ...
     641\end{verbatim}
    486642
    487643You should implement the \code{\_latex\_} and \code{\_repr\_} method
    488644for every object. The other methods depend on the nature of the
     
    491647\subsection{\Latex Representation}
    492648Every object x in \sage should support the command \code{latex(x)}, so
    493649that any \sage object can be easily and accurately displayed via
    494 \Latex.  Here is how to make a class and therefore its instances
     650\Latex.  Here is how to make a class (and therefore its instances)
    495651support the command \code{latex}.
    496652
    497653\begin{enumerate}
    498654
    499 \item Put \code{import sage.misc.latex as latex} at the top of the
    500   file that defines your class.
    501 
    502 \item Define a member function \code{_latex_(self)} that returns a
     655\item Define a method \code{_latex_(self)} that returns a
    503656  \Latex representation of your object.  It should be something that
    504657  can be typeset correctly within math mode.
    505658
    506 \item Often objects are built up out of other options.  For example if
     659\item Often objects are built up out of other \sage objects, and these
     660  components should be typeset using the \code{latex} function.
     661  For example if
    507662  \code{c} is a coefficient of your object, and you want to typeset
    508663  \code{c} using latex, use \code{latex(c)} instead of
    509664  \code{c._latex_()}, since \code{c} might not have a \code{_latex_}
     
    513668  illustrates latex generation for your object.
    514669
    515670\item You can use any macros included in \code{amsmath},
    516   \code{amssymb}, \code{amsfonts} or the ones defined in
     671  \code{amssymb}, or \code{amsfonts}, or the ones defined in
    517672  \code{SAGE_ROOT/doc/commontex/macros.tex}.
    518 
    519 \item Use \code{view(x)} to view the typeset version of an object x.
    520673
    521674\end{enumerate}
    522675%skip
    523676
    524 An example template for \Latex representation follows:
     677An example template for a \code{_latex_} method follows:
    525678
    526679\begin{verbatim}
    527 import sage.misc.latex as latex
    528 ...
    529680class X:
    530681   ...
    531682   def _latex_(self):
     
    540691       return `\\frac{%s}{%s}''%(latex(self.numer), latex(self.denom))
    541692\end{verbatim}
    542693
     694As shown in the example, \code{latex(a)} will produce \Latex code
     695representing the object \code{a}.  Calling \code{view(a)} will display the
     696typeset version of this.
     697
     698
     699
    543700\subsection{Print Representation}
    544701The standard Python printing method is \code{__repr__(self)}.  In
    545702\sage, that is for objects that derive from \code{SageObject} (which
    546 is everything in \sage), instead define \code{_repr_(self)}.  This is
    547 preferable because if you only define \code{_repr_(self)} and not
    548 \code{__repr__(self)}, then users can rename your object to print
    549 however they like.
     703is everything in \sage), instead define \code{_repr_(self)}.
     704% This is preferable because if you only define \code{_repr_(self)} and
     705% not \code{__repr__(self)}, then users can rename your object to print
     706% however they like.
    550707
    551708Here is an example of the \code{_latex_} and \code{_repr_} functions
    552709for the Pi class. It is from the file
    553 \code{SAGE_ROOT/devel/sage/sage/functions/constants.py}
     710\code{SAGE_ROOT/devel/sage/sage/functions/constants.py}:
    554711
    555 \begin{verbatim}%skip
     712%skip
     713\begin{verbatim}
    556714class Pi(Constant):
    557715    """
    558716    The ratio of a circle's circumference to its diameter.
     
    571729        return "\\pi"
    572730\end{verbatim}
    573731
    574 \subsection{Matrix from Object}
    575 Provide a \code{_matrix_} member function for an object that can be
     732\subsection{Matrix or Vector from Object}
     733Provide a \code{_matrix_} method for an object that can be
    576734coerced to a matrix over a ring $R$. Then the \sage function
    577735\code{matrix} will work for this object.
    578736
    579 The following is from \code{SAGE_ROOT/devel/sage/sage/graphs/graph.py}
     737The following is from \code{SAGE_ROOT/devel/sage/sage/graphs/graph.py}:
    580738
    581739\begin{verbatim}
    582 class SimpleGraph(GenericGraph):
    583    ...
    584    def _matrix_(self, R):
    585        return self.am()
     740class GenericGraph(SageObject):
     741    ...
     742    def _matrix_(self, R=None):
     743        if R is None:
     744            return self.am()
     745        else:
     746            return self.am().change_ring(R)
    586747
    587    def am(self):
    588        """
    589        Shorter call for adjacency matrix makes life easier.
    590        """
    591        return self.adjacency_matrix()
     748
     749    def adjacency_matrix(self, sparse=None, boundary_first=False):
     750        ...
     751
     752    am = adjacency_matrix # shorter call makes life easier
    592753\end{verbatim}
    593754
    594 \subsection{Vector from Object}
    595 Provide a \code{_vector_} member function for an object that can be
     755Similarly,
     756provide a \code{_vector_} method for an object that can be
    596757coerced to a vector over a ring $R$. Then the \sage function
    597 \code{vector} will be work for this object.
     758\code{vector} will work for this object.
    598759
    599760\todo{Provide example from a .py file
    600761
    601762  The following is from the file
    602   SAGE_ROOT/sage/sage/modules/free_module_element.pyx
     763  \code{SAGE_ROOT/sage/sage/modules/free_module_element.pyx}:
    603764
    604765\begin{verbatim}
    605766cdef class FreeModuleElement(element_Vector):   # abstract base class
     
    610771}
    611772
    612773
     774\section{\SAGE Preparsing}
     775\label{sec:Preparsing}
     776
     777The following files are relevant to preparsing in \sage:
     778
     779\begin{enumerate}
     780
     781\item \code{SAGE_ROOT/local/bin/sage-sage},
     782
     783\item \code{SAGE_ROOT/local/bin/sage-preparse},
     784
     785\item \code{SAGE_ROOT/devel/sage/sage/misc/preparser.py}
     786
     787% \item \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py}
     788
     789\todo{Talk about
     790    \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py} file}
     791
     792\end{enumerate}
     793
     794In particular, the file \code{preparser.py} contains the \SAGE
     795preparser code.  Here are some , and the following are some notes from it:
     796
     797\begin{itemize}
     798
     799\item In \SAGE, methods can be called on integer and real literals --
     800  note that in pure Python this would be a syntax error.  For example:
     801\begin{verbatim}
     802    sage: 16.sqrt()
     803    4
     804    sage: 87.factor()
     805    3 * 29
     806\end{verbatim}
     807
     808\item Raw literals are not preparsed, which can be useful from an
     809  efficiency point of view.  Just like Python ints are denoted by an
     810  L, in \SAGE raw integer and floating literals are followed by an ``r''
     811  (or ``R'') for raw, meaning not preparsed.  Example:
     812\begin{verbatim}
     813    sage: a = 393939r
     814    sage: a
     815    393939
     816    sage: type(a)
     817    <type 'int'>
     818    sage: b = 393939
     819    sage: type(b)
     820    <type 'sage.rings.integer.Integer'>
     821    sage: a == b
     822    True
     823\end{verbatim}
     824
     825\item Raw literals can be very useful in certain cases. For instance,
     826  Python integers are typically much more efficient than \SAGE integers
     827  when they are very small; large \SAGE integers are much more
     828  efficient than Python integers, since they are implemented using the
     829  GMP C library.  For example, the first of these commands may be somewhat
     830  slower than the second:
     831\begin{verbatim}
     832    sage: v = [ 2*i for i in range(10000)]
     833    sage: v = [ 2r*i for i in range(10000r)]
     834\end{verbatim}
     835because the first uses \sage integers, while the second uses raw
     836Python integers.
     837
     838\end{itemize}
     839
     840Consult the file \code{preparser.py} for more details about \SAGE
     841preparsing, more examples involving raw literals, etc.
     842
     843When a file \code{foo.sage} is loaded in a \sage session, a preparsed version of
     844\code{foo.sage} is created titled \code{foo.py}. The beginning of \code{foo.py} states:
     845\verb+This file was *autogenerated* from the file foo.sage.+
     846
     847
    613848\section{The \SAGE Rules and Conventions for Coercion and Arithmetic}
    614 \note{These are the {\bf planned} rules and conventions.  They have not
    615 been systematically implemented throughout the system yet.}
    616849
    617 Let $R$ and $S$ be \SAGE parent structures, e.g.,
    618 groups, rings, point sets, etc.\todo{Make section gentler}
     850\note{August 2008: Much of this material is out of date.  We are
     851working on a revised version.}
     852
     853\todo{Make section gentler}
     854
     855Let $R$ and $S$ be \SAGE parent structures, e.g., groups, rings, point
     856sets, etc.
    619857
    620858\begin{itemize}
    621 \item {\bf OBJECT COERCION \code{_coerce_}:}
     859\item \textbf{OBJECT COERCION \code{_coerce_}:}
    622860 Suppose a \code{_coerce_} map $R \to S$ is defined.  This
    623861is a method of $S$, so it is called using the notation \code{S._coerce_(x)}, where $R$ is a parent of $x$.
    624862It's signature is
     
    679917rules.  E.g., embeddings of finite fields via Conway polynomials, or
    680918inclusions of extensions of number fields, fit into this structure, as
    681919does the inclusion $\QQ \hookrightarrow \CC$ and the surjection $\ZZ
    682 \to \ZZ/n\ZZ$.  The function \code{_coerce_} {\bf does not} have to be
     920\to \ZZ/n\ZZ$.  The function \code{_coerce_} \textbf{does not} have to be
    683921``canonical'' in a precisely defined mathematical sense.
    684922 
    685 \item {\bf OBJECT CREATION \code{__call__}:}
     923\item \textbf{OBJECT CREATION \code{__call__}:}
    686924  If you write $R(x)$, then the \code{__call__} method of $x$ is called.  It
    687925should make an object of $R$ from $x$, if at all possible.
    688926     \code{__call__}  is never called implicitly by binary operators.
    689927    If $x$ already has $R$ as its  parent, then \code{__call__} must return
    690     a {\bf new copy} of $x$, unless $x$ is immutable (this is to agree with
     928    a \textbf{new copy} of $x$, unless $x$ is immutable (this is to agree with
    691929    standard Python conventions).
    692930For example, if $x \in \ZZ/n\ZZ$, then \code{ZZ.__call__(x)} is defined but
    693931  \code{ZZ.__coerce__(x)} is not.
    694932
    695 \item {\bf ARITHMETIC \code{__add__}, \code{__mul__}, ...:}: When
     933\item \textbf{ARITHMETIC \code{__add__}, \code{__mul__}, ...:}: When
    696934  doing a binary operation, if the parents are not identical (in the
    697935  sense of is\todo{incomplete?}), determine if precisely one {\tt
    698936    \_coerce\_} map is defined; if so, apply it and do the arithmetic
    699937  operation.  If both are defined, the parents are canonically
    700938  isomorphic, so use the left one.  If neither are defined, raise a
    701   TypeError.  (Whether or not there is a coerce map between objects \todo{the objects?}
     939  TypeError.  (Whether or not there is a coerce map between objects
    702940  should be cached for efficiency.)
    703941 
    704 \item {\bf PARENT OBJECT \code{__cmp__}}: \code{R == S} by definition
     942\item \textbf{PARENT OBJECT \code{__cmp__}}: \code{R == S} by definition
    705943  means that \code{_coerce_} is defined in both directions.  Roughly
    706944  speaking this means that $R$ and $S$ are identical or canonically
    707945  isomorphic, though there could be some exceptions to this.  Note
     
    709947  considered equal, since there are never canonical coercion maps in
    710948  both directions when the categories are different.
    711949 
    712 \item {\bf ELEMENT \code{__cmp__}}:
     950\item \textbf{ELEMENT \code{__cmp__}}:
    713951       If the parents aren't identical, test if
    714        precisely one \code{_coerce_} map is defined -- if so, return \code{__cmp__}
     952       precisely one \code{_coerce_} map is defined --- if so, return \code{__cmp__}
    715953       after applying the coerce.  If both coercions are defined, compute
    716954       both \code{__cmp__}'s (in both $R$ and $S$).  Return the value if they give
    717955       the same results.  Otherwise return $-1$ (the convention in Python
    718956       is that \code{__cmp__} never raises an exception).   If no \code{_coerce_} is
    719957       defined return $-1$ (i.e., not equal).
    720958 
    721 \item {\bf IN \code{__contains__}}:
     959\item \textbf{IN \code{__contains__}}:
    722960       \code{x in R} should be true if and only if \code{R._coerce_(x)} would not
    723961       raise a \code{TypeError}.
    724962\end{itemize}
    725963 
     964
    726965\section{Mutability}
     966
    727967Parent structures (e.g., rings, fields, matrix spaces, etc.) should
    728968be immutable and globally unique whenever possible.   Immutability
    729 includes making it so properties like generator labels and default
     969means, among other things, that properties like generator labels and default
    730970coercion precision cannot be changed.
    731971
    732 Global uniqueness while not wasting space storing objects that are not
    733 being referenced is best implemented using the standard Python weakref
     972Global uniqueness while not wasting memory
     973is best implemented using the standard Python weakref
    734974module, a factory function, and module scope
    735 variable.\todo{Rewrite. Difficult to parse. Make gentler}
     975variable.
     976\todo{Rewrite. Difficult to parse. Make gentler}
    736977
    737 \todo{Put a tutorial on this here}.
     978\todo{Put a tutorial on this here}
    738979
    739980Certain objects, e.g., matrices, may start out mutable and become
    740981immutable later.  See the file \code{SAGE_ROOT/devel/sage/sage/structure/mutability.py}.
    741982
    742 \section{Importing Modules}
    743 Do not import at the top level of your module a third-party module that
    744 will take a long time to initialize (e.g., matplotlib).  It is important
    745 that \code{from sage.all import *} not take forever, since this is what
    746 dominates the \SAGE startup time. 
    747 
    748 On a somewhat regular basis I do the following:
    749 \begin{verbatim}
    750   echo "%prun import sage.all" | sage -ipython|more
    751 \end{verbatim}
    752 in order to profile what is taking a long time during
    753 the load.  In many cases I use the locate command to
    754 find files mentioned in the profile.
    755983
    756984\section{The {\tt \_\_hash\_\_} Special Method}
     985
    757986Here's the definition of \code{__hash__}
    758987from the Python reference manual.
    759988
     
    7801009Python language, {\em which in \sage we simply cannot make} (!), and
    7811010violating it has consequences.  Fortunately, the consequences are pretty
    7821011clearly defined and reasonably easy to understand, so if you know
    783 about them they don't cause you trouble.  I think the following
     1012about them they don't cause you trouble.  The following
    7841013example illustrates them pretty well:
    7851014\begin{verbatim}
    7861015    sage: v = [Mod(2,7)]
     
    8001029    KeyError: 9
    8011030\end{verbatim}
    8021031
    803 Here's another example,
     1032Here's another example:
    8041033\begin{verbatim}
    8051034    sage: R = RealField(10000)
    8061035    sage: a = R(1) + R(10)^-100
     
    8111040
    8121041Unfortunately, in \SAGE we simply can't require:
    8131042\begin{verbatim}
    814           (*) "a == b ==> hash(a) == hash(b)"
     1043       (*)  "a == b ==> hash(a) == hash(b)"
    8151044\end{verbatim}
    8161045because serious mathematics is simply too complicated for this
    8171046rule.    For example, the equalities
     
    8201049integers.
    8211050
    8221051The only way we could ``fix'' this problem for good would be to
    823 abandon using the "==" operator for "\SAGE equality", and implement
     1052abandon using the \code{==} operator for ``\SAGE equality'', and implement
    8241053\SAGE equality as a new method attached to each object. Then we could
    825 follow Python rules for "==" and our rules for everything else, and
     1054follow Python rules for \code{==} and our rules for everything else, and
    8261055all \SAGE code would become completely unreadable (and for that matter
    8271056unwritable).  So we just have to live with it.
    8281057
    829 So what is done in \sage is to {\em attempt} to satisfy (*) when it
     1058So what is done in \sage is to {\em attempt} to satisfy \verb+(*)+ when it
    8301059is reasonably easy to do so, but use judgment and not go overboard.
    8311060For example,
    8321061\begin{verbatim}
    8331062    sage: hash(Mod(2,7))
    8341063    2
    8351064\end{verbatim}
    836 I.e., we get 2.  That's better than some weird random hash that also
     1065The output 2 is better than some random hash that also
    8371066involves the moduli, but it's of course not right from the Python point
    8381067of view, since \code{9 == Mod(2,7)}.
    8391068
    8401069The goal is to make a hash function that is fast but within reason
    841 respects any obvious, natural inclusions and coercions.
     1070respects any obvious natural inclusions and coercions.
    8421071
    8431072
     1073\section{Exceptions}
    8441074
    845 
    846 
    847 \chapter{Creating a \SAGE Module or Package}
    848 
    849 \todo{What is a \SAGE Module and what is a \SAGE Package? Are they the
    850   same? This chapter talks about only the latter? When
    851   should/shouldn't one make a package out of a bunch of files?}
    852 
    853 \begin{enumerate}
    854 \item Design: Think about what your program will do and how that fits into the
    855   structure of \SAGE.  In particular, much of \SAGE is implemented in
    856   the object-oriented language Python, and there is a hierarchy of
    857   classes that organize code and functionality.  It is {\em
    858     surprisingly painful and difficult} to create some Python classes,
    859   then realize that much functionality should be factored out into
    860   base classes, and to do it\todo{Not sure what you are trying to
    861     convey here}.  So try to get the structure right the first time.
    862   For example, if you implement elements of a ring your class should
    863   derive from
    864   \class{sage.structure.element.RingElement}.\todo{Design is being
    865     talked about without using the word. Consciously avoiding the
    866     term?}
    867 
    868 \item Coding: Code it up, following the conventions in
    869   Chapter~\ref{ch:conventions}.  Make the basic structure and
    870   interface for what you're doing in Python.  You can use any of the
    871   following languages to implement the hard parts: Python, C/C++,
    872   Cython, Fortran 95, GAP, Singular, and GP/PARI.  You can also use the mwrank,
    873   GSL, NTL, and PARI C/C++ libraries. (And if you are OK with your code
    874   depending on optional \sage packages, you can use Octave, or
    875   even Magma, Mathematica, or Maple.)  For readability and usability
    876   it is best to use \sage when possible. On the other hand, for
    877   efficiency reasons it is often crucial to use a compiled language or
    878   an existing library or program---do not reinvent the wheel or write
    879   code that is so slow as to be only a toy\todo{I split the
    880     original line into two, as reusing existing code should be
    881     emphasized more}.
    882 
    883 \item Documentation: Document {\em every} Python function that you've
    884   defined,
    885 
    886 \item Testing: Liberally include examples in your source code. See
    887   Chapter~\ref{ch:testing} for more on automated testing of examples.
    888 
    889 \item (Optional) Write a file that generates random input to test (and
    890   break!) your code with the intention of making it robust (See
    891   Chapter~\ref{ch:randomtesting}).
    892 
    893 \item (Optional) Send your package to the \code{sage-devel} Google group
    894   for inclusion in \SAGE, or make a standalone
    895   {\tt spkg} (see Chapter~\ref{ch:spkg}).
    896 
    897 \end{enumerate}
    898 
    899 
    900 \section{\SAGE Preparsing}
    901 \label{sec:Preparsing}
    902 
    903 The following is a list of files relevant to preparsing through which
    904 control flows when \sage is run:
    905 
    906 \begin{enumerate}
    907 
    908 \item \code{SAGE_ROOT/sage-sage},
    909 
    910 \item \code{SAGE_ROOT/sage-preparse},
    911 
    912 \item \code{SAGE_ROOT/devel/sage/sage/misc/preparser.py}
    913 
    914   \todo{Talk about
    915     \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py} file}
    916 
    917 \end{enumerate}
    918 
    919 The file preparser.py is contains the \SAGE preparser code and the
    920 following are some notes from the aforementioned file
    921 
    922 \begin{itemize}
    923 
    924 \item In \SAGE methods can also be called on integer and real literals
    925   (note that in pure Python this would be a syntax error).
    926 
    927 \item Note that calling methods on int literals in pure Python is a
    928   syntax error, but \SAGE allows this for \SAGE integers and reals,
    929   because users frequently request it.
    930 
    931 \item Raw literals are not preparsed, which can be useful from an
    932   efficiency point of view.  Just like Python ints are denoted by an
    933   L, in \SAGE raw integer and floating literals are followed by an ``r''
    934   (or ``R'') for raw, meaning not preparsed.
    935 
    936 \item Raw literals can be very useful in certain cases. For instance,
    937   Python integers are typically much more efficient than \SAGE integers
    938   when they are very small; large \SAGE integers are much more
    939   efficient than Python integers, since they are implemented using the
    940   GMP C library --- see section \ref{sec:Benchmarking}.
    941 
    942 \end{itemize}
    943 
    944 
    945 Consult preparser.py for details about \SAGE preparsing, examples involving raw literals, etc.
    946 
    947 When foo.sage is loaded in a sage session, a preparsed version of
    948 foo.sage is created titled foo.py. The beginning of foo.py states:
    949 \code{This file was *autogenerated* from the file foo.sage.}
    950 
    951 \section{Randomized Testing}
    952 \label{ch:randomtesting}
    953 In addition to all the examples, which serve as both demonstrations
    954 and tests of your code, you should also create a test suite.  Think of
    955 this as a program that will run for a while and ``tries'' to crash
    956 your code using randomly generated input.  Your test code should
    957 define a class \class{Test} with a {\tt random()} method that runs
    958 random tests.  These are all assembled together later, and each test
    959 is run for a certain amount of time on a regular basis.
    960 
    961 \section{Creating a New \sage spkg Package}
    962 \label{ch:spkg}
    963 Here's how to make your own \sage package.
    964 
    965 \begin{itemize}
    966 \item Note that a \sage package is simply a .tar.bz2 file (or a tar
    967   file), but named with .spkg to discourage confusion.  In particular,
    968   you can type
     1075Please avoid code like this:
    9691076%skip
    9701077\begin{verbatim}
    971             tar jxvf my_package-version.spkg
     1078try:
     1079    some code
     1080except:               # bad
     1081    more code
    9721082\end{verbatim}
    973 to extract one and see what is inside.
     1083Instead catch specific exceptions. For example,
     1084\begin{verbatim}
     1085try:
     1086    return self.__coordinate_ring
     1087except (AttributeError, other exceptions), msg:           # Good
     1088    more code to compute something
     1089\end{verbatim}
     1090Note that the syntax in \code{except} is to list all the exceptions
     1091that are caught as a tuple, followed by an error message.
    9741092
    975 \item
    976 To create your own package:
    977 \begin{itemize}
    978 \item[(a)]
    979 Make a directory, e.g., \code{mypackage-0.1}.
    980 The name of the directory should be a lower-case string with no
    981 dashes followed by a dash followed by a version number.
     1093If you don't have any exceptions explicitly listed (as a tuple), your
     1094code will catch absolutely anything, including \code{ctrl-C} and alarms,
     1095and this will lead to confusion.
    9821096
    983 \item[(b)]
    984 Put your files in that directory.
    9851097
    986 \item[(c)]
    987 Create an executable shell script \code{mypackage-0.1/spkg-install}.
     1098\section{Importing}
    9881099
    989 \item[(d)]
    990 The script \code{spkg-install} is run during installation
    991 of the \sage package.   In this script you may
    992 make the following assumptions:
    993 \begin{itemize}
    994 \item
    995 The PATH has \code{sage} and the \sage install's \code{python}
    996 location at the front.  Thus the command
     1100We mention two issues with importing: circular imports and importing
     1101large third-party modules.
     1102
     1103First, you must avoid circular imports.  For example, suppose that the
     1104file \code{SAGE_ROOT/devel/sage/sage/algebras/steenrod_algebra.py}
     1105started with a line
    9971106%skip
    9981107\begin{verbatim}
    999                      python setup.py install
     1108from sage.sage.algebras.steenrod_algebra_bases import *
    10001109\end{verbatim}
    1001 will run the correct version of Python with
    1002 everything setup correctly.  Also, e.g.,
    1003 running \code{gap} or \code{Singular} will run the
    1004 correct versions.
    1005 
    1006 \item
    1007 The environment variable \code{SAGE_ROOT} points to
    1008 the root directory of the \sage install.
    1009 \item
    1010 The environment variable \code{SAGE_LOCAL} points to
    1011 the \code{SAGE_ROOT/local} directory of the \sage install.
    1012 \item
    1013 The environment variables
    1014 \code{LD_LIBRARY_PATH} and \code{DYLD_LIBRARY_PATH}
    1015 both have \code{SAGE_ROOT/local/lib} at the front.
    1016 
    1017 \end{itemize}
    1018 
    1019 \item[(e)]
    1020 The \code{spkg-install} script should copy your files to the appropriate
    1021 place after doing any build that is necessary.  That's it!
    1022 
    1023 \item[(f)] (Optional) Send a copy to the \code{sage-devel} Google group
    1024 so it can be posted to the \SAGE web site, so anybody can automatically
    1025 install it by typing \code{sage -i my_package-version.spkg}.
    1026 \end{itemize}
    1027 \end{itemize}
    1028 
    1029 For example, the location of GAP is \code{SAGE_ROOT/local/lib/gap-4.4.6/}.
    1030 Thus to install gap code you have to copy it there.  You should check
    1031 that the directory with the given version your package is created for
    1032 exists in your \code{spkg-install} script, e.g., with the following code:
     1110and that the
     1111file \code{SAGE_ROOT/devel/sage/sage/algebras/steenrod_algebra_bases.py}
     1112started with a line
    10331113%skip
    10341114\begin{verbatim}
    1035 if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.6 ]; then
    1036     echo "This package requires that GAP 4.4.6 is installed."
    1037     exit 1
    1038 fi
     1115from sage.sage.algebras.steenrod_algebra import SteenrodAlgebra
     1116\end{verbatim}
     1117This sets up a loop: loading one of these files requires the other,
     1118which then requires the first, etc.
     1119
     1120With this set-up, running \sage will produce an error:
     1121%skip
     1122\begin{verbatim}
     1123Exception exceptions.ImportError: 'cannot import name SteenrodAlgebra' in 'sage.rings.polynomial.polynomial_element.Polynomial_generic_dense.__normalize' ignored
     1124---------------------------------------------------------------------------
     1125ImportError                               Traceback (most recent call last)
     1126
     1127...
     1128ImportError: cannot import name SteenrodAlgebra
     1129\end{verbatim}
     1130Instead, you might replace the \code{import *} line at the top of the
     1131file by more specific imports where they are needed in the code
     1132-- e.g., the \code{basis} method for the class \code{SteenrodAlgebra}
     1133might look like this (omitting the documentation string):
     1134%skip
     1135\begin{verbatim}
     1136    def basis(self, n):
     1137        from steenrod_algebra_bases import steenrod_algebra_basis
     1138        return steenrod_algebra_basis(n, basis=self._basis_name, p=self.prime)
    10391139\end{verbatim}
    10401140
    1041 \emph{Caveat}: Do {\em not} just
    1042 copy to \code{SAGE_ROOT/local/lib/gap*/} since that will copy
    1043 your package to the lib directory of the {\em old} version of GAP if
    1044 GAP is upgraded.
     1141Second, do not import at the top level of your module a third-party
     1142module that will take a long time to initialize (e.g., matplotlib).
     1143As above, you might instead import specific components of the module
     1144when they are needed, rather than at the top level of your file.
    10451145
    1046 External Magma code goes in
    1047 \verb+SAGE_ROOT/data/extcode/magma/user+, so if you want to redistribute
    1048 Magma code with \sage as a package that Magma-enabled users can use,
    1049 that's where you'd put it.  You'd also want to have relevant Python code
    1050 to make the Magma code easily usable.
     1146It is important to try to make \code{from sage.all import *} as fast
     1147as possible, since this is what dominates the \SAGE startup time, and
     1148controlling the top-level imports helps to do this.
    10511149
    1052 {\bf Example}:
    1053 These instructions provide directions for creating a particular
    1054 \sage package. \todo{Something about this example bugs me and I'm not sure what it is.}
    10551150
    1056 \begin{itemize}
    1057 \item
    1058 Download \code{guava2.4.tar.gz} (the tarball for GUAVA, the
    1059 GAP error-correcting codes package) from
    1060 \code{http://www.gap-system.org/Packages/guava.html}.
     1151\section{Using Optional Packages}
    10611152
    1062 \item
    1063 Suppose you have a directory \code{sage-package-guava}.
    1064 Within that, create a directory \code{guava-2.4}.
    1065 Extract the tarball into that directory
    1066 (so there will be a \code{guava2.4} subdirectory of
    1067 \code{sage-package-guava}).
     1153If a function requires an optional package, that function should fail gracefully ---
     1154perhaps using a \code{try} and \code{except} block --- when the optional
     1155package is not available, and should give a hint about how to install
     1156it.  For example, typing \code{sage -optional} gives a list of all
     1157optional packages, so it might suggest to the user that they type that.
    10681158
    1069 \item
    1070 Put the following script called \code{spkg-install} in
    1071 \code{sage-package-guava}:
    10721159
    1073 %skip
    1074 \begin{verbatim}
    1075 #!/bin/sh
    10761160
    1077 if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.6 ]; then
    1078     echo "This package requires that GAP 4.4.6 is installed."
    1079     exit 1
    1080 fi
     1161\chapter{Coding in Other Languages}
    10811162
    1082 cp -pr guava2.4 $SAGE_ROOT/local/lib/gap-4.4.6/pkg/
     1163When writing code for \sage, use Python for the basic structure and
     1164interface.  For speed, efficiency, or convenience, you can implement
     1165parts of the code using any of the following languages:
     1166Cython, C/C++, Fortran 95, GAP, Singular, and GP/PARI.  You can also
     1167use the mwrank, GSL, NTL, and PARI C/C++ libraries. (And if you are
     1168okay with your code depending on optional \sage packages, you can use
     1169Octave, or even Magma, Mathematica, or Maple.)
    10831170
    1084 cd $SAGE_ROOT/local/lib/gap-4.4.6/pkg/guava2.4
    1085 ./configure ../..
    1086 make
    1087 \end{verbatim}
     1171The first section of this chapter discusses Cython (which is a
     1172compiled language, based on Python); many components of \sage are
     1173written in Cython.  Later sections discuss the interfaces between
     1174\sage and PARI, GAP, and Singular.
    10881175
    1089 Make it executable with \code{chmod +x spkg-install}.
    10901176
    1091 \item
    1092 In the directory \code{sagefiles},
    1093 issue the command
    10941177
    1095 %skip
    1096 \begin{verbatim}
    1097 <path>sage -pkg guava-2.4
    1098 \end{verbatim}
    1099 where \verb+<path>+ means the absolute path to the sage
    1100 script. This will do some checks, run tar/bz2, and create the spkg file,
    1101 \code{guava-2.4.spkg}, in the
    1102 directory \code{sagefiles},
    1103 \end{itemize}
     1178\section{Cython}
    11041179
    1105 To test this, within the \code{sagefiles} directory
    1106 (or whatever directory contains \code{guava-2.4.spkg}),
    1107 type
     1180Cython is a compiled version of Python; it is based on Pyrex
     1181(\url{http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/}).  To
     1182a large degree, Cython has changed based on what \sage's developers
     1183needed: Cython has been developed in concert with \sage.
    11081184
    1109 %skip
    1110 \begin{verbatim}
    1111 <path>sage -i guava-2.4.spkg
    1112 \end{verbatim}
    1113 It should install and compile the guava program in sage.
    1114 To test its functionality, one could do the following:
     1185As such, it is a young, but developing, language, with young, but
     1186developing, documentation.  See its web page,
     1187\url{http://www.cython.org/}, for the most up-to-date information.
    11151188
    1116 \begin{verbatim}
    1117 sage: gap(`LoadPackage("guava")')          # requires optional package
    1118 true
    1119 sage: gap(`HammingCode(3,3)')              # requires optional package
    1120 a linear [13,10,3]1 Hamming (3,3) code over GF(3)
    1121 \end{verbatim}
    1122 
    1123 To  build again if you've already installed the package, use
    1124 %skip
    1125 \begin{verbatim}
    1126 <path>sage -f guava-2.4.spkg
    1127 \end{verbatim}
    1128 
    1129 
    1130 \chapter{Documentation}
    1131 
    1132 \section{Source Code Documentation}
    1133 Use \Latex formatting in the documentation string.  If you wish to use
    1134 backslashes in the documentation string, use double backslashes or place an {\tt r} right before the first triple opening quote. For example,
    1135 %skip
    1136 \begin{verbatim}
    1137 def cos(x):
    1138     """
    1139     Returns $\\cos(x)$.
    1140     """
    1141 
    1142 def sin(x):
    1143     r"""
    1144     Returns $\sin(x)$.
    1145     """
    1146 \end{verbatim}
    1147 
    1148 The non-standard macros that are used when typesetting the
    1149 documentation are defined in
    1150 \code{SAGE_ROOT/doc/commontex/macros.tex}.  If you need to add more macros,
    1151 add them there and post them on the \code{sage-devel} Google group so that
    1152 they can be included in the next version of \sage.
    1153 
    1154 
    1155 The first line two or three lines of the docstring should briefly summarize
    1156 what the function does, e.g., ``Returns X'' or ``Computes X'',
    1157 or ``Creates a new X''.  Next describe the input and output of
    1158 the function using the following notation:
    1159 %skip
    1160 \begin{verbatim}
    1161 INPUT:
    1162     var1 -- <type> (default: ...) description
    1163     var2 -- <type> (default: ...) description
    1164     etc.
    1165 
    1166 OUTPUT:
    1167     <type> -- description of return value 1
    1168     <type> -- description of return value 2
    1169     etc.
    1170 \end{verbatim}
    1171 
    1172 This is a placeholder of a "Perfect example of the above".\todo{
    1173 I=ifti propose everywhere in this document when there is a template, it is followed by a path/filename where in one can find a "perfect" example.}
    1174 
    1175 For consistency you should use the above format, which is used
    1176 throughout the system.  Do not just describe the input and output
    1177 in long sentences.    For methods of a class, whether or not
    1178 to describe self is up to you; this can be useful if self must
    1179 have some special properties.  Also, you do not have to
    1180 specify the types of the input and output arguments
    1181 precisely; e.g., use ``integer'' for an int or long or
    1182 \sage \code{Integer}.
    1183 
    1184 Every function should have at least one, and preferably many, examples
    1185 that illustrate its usage. These examples must exactly match the
    1186 output of an actual running copy of Python.  To test whether this is
    1187 the case, run the program \code{sage -t} with your Python or Cython
    1188 file as unique argument (see Chapter~\ref{ch:testing} for more details).
    1189 
    1190 For more details, see Section~\ref{sec:conv-docs}.\todo{redundancy \S 2.4 and \S4.1}
    1191 
    1192 \section{Extra Documentation}
    1193 You might also write additional documentation in \Latex, which is not to
    1194 be part of any source code file.  The examples in this documentation
    1195 should be contained in verbatim environments.  The examples should be
    1196 tested using \code{sage-testtex}, which you run on the \Latex file.  Put
    1197 \%skip on the line preceding the environment to skip testing of an
    1198 example.
    1199 
    1200 Note that \code{sage-testtex} is completely different than \code{sage-doctest}.  It extracts the documentation and feeds it to a
    1201 running instance of \SAGE.
    1202 
    1203 
    1204 \section{Automated Testing}\label{ch:testing}
    1205 This section describes automated testing of test files of the
    1206 following types: \code{.py .pyx .sage .tex}.  Use \code{sage -t
    1207   <file>} to test that the examples in \code{<file>} behave exactly as
    1208 claimed.  See Section~\ref{sec:conv-docs} for a discussion of how to
    1209 include examples in documentation strings and what conventions to
    1210 follow --- Section ~\ref{sec:Further-conventions}.
    1211 
    1212 \subsection{Testing .py, .pyx and .sage Files}\label{sec:test:python}
    1213 Run \code{sage -t <filename.py>} to test that all code examples in
    1214 \code{filename.py}.  Similar remarks apply to \code{.sage} and
    1215 \code{.pyx} files.
    1216 
    1217 \begin{verbatim}
    1218   sage -t [--verbose] [--optional]  [files and directories ... ]
    1219 \end{verbatim}
    1220 
    1221 When you run \code{sage -t <filename.py>}, \sage makes a copy of
    1222 \code{<filename.py>} with all the \code{sage} prompts replaced by
    1223 \code{{>}{>}>}, then uses the standard Python doctest framework to test
    1224 the documentation.  More precisely,
    1225 the Python script \code{local/bin/sage-doctest} implements
    1226 documentation testing.  It does the following when asked
    1227 to test a file \code{foo.py} or \code{foo.sage}.
    1228 \begin{enumerate}
    1229 
    1230 \item Creates the directory \code{.doctest} if it doesn't
    1231 exist and the file \code{.doctest/foo.py}.
    1232 
    1233 \item The file \code{.doctest_foo.py} contains functions for
    1234 each docstring in \code{foo.py}, but with all \sage preparsing
    1235 applied and with \code{from sage.all import *} at the top.
    1236 The function documentation is thus standard Python with
    1237 \code{{>}{>}>} prompts.
    1238 
    1239 \item \code{sage-doctest} then runs \sage's Python interpreter
    1240 on \code{.doctest_foo.py}. 
    1241 \end{enumerate}
    1242 
    1243 \note{{\em Note that line numbers in the errors 
    1244 you might see apply to the file \code{.doctest_foo.py}, not
    1245 to the original file \code{foo.py}!}  If you get errors,
    1246 feel free to inspect \code{.doctest_foo.py}; it is never
    1247 automatically deleted by \sage.}
    1248 
    1249 Your file passes these tests if the code in it will run when entered
    1250 at the \code{sage:} prompt with no special imports.  Thus users are
    1251 guaranteed to be able to exactly copy code out of the examples you
    1252 write for the documentation and have them work.
    1253 
    1254 %skip
    1255 \begin{verbatim}
    1256 
    1257 William:
    1258 
    1259 I changed the doctest behavior recently so that now you can doctest
    1260 files anywhere (not necessary in \SAGE), and type e.g.,
    1261 
    1262     sage: from foo import *
    1263     sage: ...
    1264 
    1265 in the doctest blocks and it will work, even if foo.py isn't part of the
    1266 \SAGE library.  It's much better now.   Feel free to change the reference
    1267 manual to reflect this.
    1268 
    1269 \end{verbatim}
    1270 
    1271 
    1272 \subsection{Testing \Latex Documentation}
    1273 
    1274 Run \code{sage -t <filename.tex>} to test the examples in
    1275 verbatim environments in \Latex documentation.   Put
    1276 \code{\%skip}
    1277 on the line right before a verbatim environment to have that
    1278 example skipped when testing the file.   
    1279 
    1280 \sage creates a file \code{.doctest_filename.py} and tests
    1281 it just as for \code{.py}, \code{.pyx} and \code{.sage}
    1282 files.   In order to skip testing of a block of code
    1283 in a verbatim environment, put the \Latex comment
    1284 \code{\%skip} on the previous line.
    1285 
    1286 One difference is that in \Latex files one
    1287 often inserts explanatory texts between different
    1288 verbatim environments.    To link together verbatim
    1289 environments use the \code{\%link} comment.  For
    1290 example\todo{I have changed verbatim to verbatim0 here
    1291 since I don't know how to include end{verbatim} in a
    1292 verbatim environment.}
    1293 %skip
    1294 \begin{verbatim}
    1295     \begin{verbatim}
    1296     sage: a = 1
    1297     \end{verbatim_}%link
    1298 
    1299     Next we add 1 to \code{a}.
    1300     %link
    1301     \begin{verbatim}
    1302     sage: 1 + a
    1303     2
    1304     \end{verbatim_}
    1305 \end{verbatim}
    1306 
    1307 See \code{SAGE_ROOT/doc/tut/tut.tex} for many examples
    1308 of how to include automated testing in \Latex documentation
    1309 for \sage.
    1310 
    1311 
    1312 
    1313 \section{Modifying the \SAGE Manuals}
    1314 We use the tutorial as an example.
    1315 
    1316 You just modify the file \code{SAGE_ROOT/devel/doc/tut/tut.tex} To
    1317 build a test copy of the tutorial with your changes, type
    1318 \code{./build_dvi} in the \code{devel/doc/tut} directory.  You'll get
    1319 a file \code{tut.dvi}.
    1320 You can also build HTML versions of everything by typing \code{make
    1321   html} in the \code{devel/doc} directory.  Build pdfs with \code{make
    1322   pdf}. \todo{You could also do \code{./build_pdf} to create a pdf directly from the tex file.}
    1323 
    1324 It is a good idea to type \code{sage -t tut.tex} to make sure any
    1325 examples you added to the tutorial work as claimed.
    1326 
    1327 The \code{SAGE_ROOT/devel/doc} directory is a Mercurial repository
    1328 (see Chapter~\ref{ch:mercurial}), so you can see exactly what changes
    1329 you've made to the documentation, make patches of what you've done
    1330 available, etc.  You can do this from within \sage by typing
    1331 \code{hg_doc.[tab]}.
    1332 
    1333 Imagine you have edited this Programming Manual found in \code{devel/doc/prog}.
    1334 Before you started you did a pull to be sure your docs are up to date:
    1335 \todo{Give some examples of usage}
    1336 
    1337 %skip
    1338 \begin{verbatim}
    1339 sage: hg_doc.pull()
    1340 cd "/home/jaap/sage/devel/doc" && hg status
    1341 cd "/home/jaap/sage/devel/doc" && hg status
    1342 cd "/home/jaap/sage/devel/doc" && hg pull -u http://sagemath.org/sage//hg//doc-main
    1343 pulling from http://sagemath.org/sage//hg//doc-main
    1344 searching for changes
    1345 adding changesets
    1346 adding manifests
    1347 adding file changes
    1348 added 3 changesets with 3 changes to 1 files
    1349 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
    1350 If it says use 'hg merge' above, then you should
    1351 type hg_sage.merge(), where hg_sage is the name
    1352 of the repository you are using.  This might not
    1353 work with the notebook yet.
    1354 \end{verbatim}
    1355 
    1356 After you made many changes, you want to commit them:
    1357 %skip
    1358 \begin{verbatim}
    1359 sage: hg_doc.commit()
    1360 cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg diff  | less
    1361 cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg commit
    1362 \end{verbatim}
    1363 
    1364 You saw the differences piped through \code{diff}, you were asked
    1365 to comment: type 'i' or 'a', your comment, hit 'Escape' and type ':wq".
    1366 
    1367 Now you can make a patch 'patchname'. e.g.:
    1368 %skip
    1369 \begin{verbatim}
    1370 sage: hg_doc.send('prog20070118.hg')
    1371 Writing to /home/jaap/sage/prog20070118.hg
    1372 cd "/home/jaap/sage/devel/doc" && hg bundle  tmphg http://sagemath.org/sage//hg//doc-main
    1373 searching for changes
    1374 Successfully created hg patch bundle /home/jaap/sage/prog20070118.hg
    1375 \end{verbatim}
    1376 
    1377 Send the file 'patchname' to the \code{sage-devel} Google group.
    1378 
    1379 
    1380 \chapter{Writing Optimized Compiled Code With \SAGE}
    1381 This chapter is about how to write fast compiled optimized code for
    1382 \sage.  It was written mainly by William Stein with contributions
    1383 from David Harvey, Martin Albrecht, David Joyner, Alex Clemesha.
    1384 Some of the text is closely copied or translated (to mathematical
    1385 language) from Greg Ewing's original Pyrex manual.
    1386 
    1387 This chapter assumes that you have a C compiler suite (e.g., GCC)
    1388 installed on your computer, since you should not write compiled
    1389 extension code on a computer without a C compiler.  If you are using
    1390 OS X, download and install the latest version of XCode.\footnote{The
    1391   download is about 1GB, and you must sign up for the Apple Developer
    1392   Network. \url{http://developer.apple.com/tools/xcode/}}
    1393 For Linux, use
    1394 your package manager to install GCC. If you're using Microsoft
    1395 Windows, then you should be using \sage via colinux, which includes
    1396 GCC by default.
    1397 
    1398 
    1399 %[[General discussion of when to use compiled code versus interpreted
    1400 %code.]]
    1401 
    1402 Interpreted code: way easier to debug,
    1403 trace, document, structure (multiple inheritence), etc.;
    1404 good when few for loops; lots of structure and calls
    1405 to compiled code.
    1406 
    1407 Compiled code: two reasons to use---make fast code and
    1408 make optimal use of libraries written in a compiled language;
    1409 hard to write and debug; takes longer to
    1410 write.  The methods explained in this chapter can produce
    1411 code where speed is only limited by your choice of algorithm.
    1412 
    1413 
    1414 
    1415 {\bf TODO:}
    1416 \begin{enumerate}
    1417 \item Module name and file name must agree!
    1418 %[(Check this at build time?)]
    1419 
    1420 \end{enumerate}
    1421 
    1422 \section{Introduction}
    1423 This chapter describes how to write \SAGE programs using Cython, which
    1424 is a language for writing Python extension modules that is a very
    1425 slightly variant of Pyrex (we will often refer to it as ``Pyrex''
    1426 still).
    1427 
    1428 The primary goal of this chapter is to provide excellent documentation
    1429 about how to use Cython to write very fast code in the context of
    1430 \SAGE.  (Currently it is a mess.)  This chapter is self contained in
    1431 that the reader should not have to look at any of the documentation
    1432 distributed with Pyrex.  The reader is urged to at some point read the
    1433 Python/C API Reference manual.
    1434 
    1435 There is also much to be learned from the Pyrex
    1436 mailing list:
    1437 
    1438 \url{http://lists.copyleft.no/mailman/listinfo/pyrex}.
    1439 
    1440 {\bf Acknowledgement:} The existence of Greg Ewing's Pyrex is
    1441 one of the primary reasons Stein chose Python as the implementation
    1442 language for \SAGE.  It is a brilliantly useful tool.
    1443 
    1444 %\cite{pyrex:list}. 
    1445 
    1446 %A list of quality open source projects that
    1447 %use Pyrex: soya3d, ... [[add more here]]
    1448 
    1449 
    1450 \subsection{Interpreted and Compiled Code}
    1451 Every serious computer algebra system, e.g., MAGMA, PARI, Mathematica,
    1452 Maple, GAP, Singular, etc., is implemented as a combination of
    1453 compiled and interpreted code.  For Mathematica, Singular, MAGMA, and
    1454 PARI, most of the implementation is in compiled C (or C++) code; some
    1455 of these systems tend to be very optimized (subject to the constraints
    1456 of the algorithms they implement). In contrast, Maple and GAP have a
    1457 relatively small compiled ``kernel'' that defines the underlying
    1458 programming language; most of the system is then implemented in this
    1459 language.  If you do benchmarks you'll discover that Mathematica is
    1460 {\em much} faster than Maple at some basic operations.  Likewise,
    1461 benchmarks reveal that MAGMA is often faster than GAP.
    1462 %[[give
    1463 %examples that illustrate each of these points.  E.g., basic
    1464 %arithmetic where maple is pitiful, computation of Bernoulli
    1465 %numbers where PARI crushes all others.]]
    1466 
    1467 This fusion of interpreted and compiled code is extremely natural for
    1468 mathematics software; some algorithms are much better implemented in
    1469 an interpreter because all time-critical steps involve low level
    1470 arithmetic --- other algorithms, e.g., matrix multiplication, must be
    1471 implemented in a compiled language in order to perform optimally.
    1472 Also, existing compiled libraries are sometimes of very high quality,
    1473 and a compiled language is needed to create the best possible
    1474 interface to such libraries.  It's crucial that both approaches to
    1475 programming be fully supported.  When deciding how to implement \SAGE,
    1476 I searched for months for an environment that could support both
    1477 approaches.  Python and Cython provide exactly this in a way that I
    1478 believe is much easier conceptually than the implementation models of
    1479 any of the other systems mentioned above.
    1480 
    1481 \subsection{Why Cython is the Best Available Option for \SAGE}
    1482 The software listed in this section is used either
    1483 to speed up Python programs or to make code written in
    1484 C++, C, FORTRAN, etc., available in Python.   This section briefly
    1485 explains what each tool does and why it is {\em not} included
    1486 in \SAGE or used in the implementation of the \SAGE library.
    1487 
    1488 
    1489 \subsubsection{SWIG}
    1490 SWIG (Standard Wrapper and Interface Generator)
    1491 takes C++ (and C, etc.) libraries, and generates
    1492 wrapper code so that the code can be used from
    1493 within Python (and in several other interpreters).  One
    1494 writes interface files that describe how to make Python
    1495 objects usable from your C++ code, and conversely what
    1496 transformations C++ objects should undergo to be usable from Python.
    1497 
    1498 In 2004, Stein considered using SWIG for \SAGE extensively, and implemented
    1499 several basic types using this model.  The idea was to
    1500 write code in C++ for \SAGE that needed to be fast, then wrap it in
    1501 SWIG.  This ground to a halt, because the result was not
    1502   {\em sufficiently} fast.  First, there is overhead when writing code
    1503 in C++ in the first place.  Second, SWIG generates several layers of
    1504 code between Python and the code that does the actual work; for
    1505 implementing any of \SAGE's numerous basic arithmetic types,
    1506 e.g., integers, the overhead is completely unacceptable, at
    1507 least if one wants code that is as fast as Magma and PARI.  For
    1508 example, in my initial implementation of integer arithmetic using
    1509 SWIG and GMP, there were the following {\em six} levels between the
    1510 \sage interpreter and the GMP C library:
    1511 \begin{center}
    1512 \begin{tabular}{|l|}\hline
    1513 Python code to provide a clean interface\\\hline
    1514 SWIG Autogenerated Python code \\\hline
    1515 SWIG Autogenerated C++ extension code \\\hline
    1516 Handcode C++ Integer class\\\hline
    1517 GMP's C++ Interface\\\hline
    1518 GMP C Library\\\hline
    1519 \end{tabular}
    1520 \end{center}
    1521 In this model, just multiplying two integers is extremely complicated,
    1522 and slow.  Moreover, writing the SWIG interface files was
    1523 painful, since general Python types are difficult to work with from
    1524 C++ code.  In contrast, with Cython the situation is as follows
    1525 (see {\tt integer.pyx}):
    1526 \begin{center}
    1527 \begin{tabular}{|l|}\hline
    1528 Cython generated C code that provides everything needed\\\hline
    1529 GMP C Library\\\hline
    1530 \end{tabular}
    1531 \end{center}
    1532 Another important problem with using SWIG is that
    1533 developers have to know C++ well and have the mental dexterity
    1534 to easily switch back and forth between C++ and Python. 
    1535 Stein does not have this ability.
    1536 
    1537 \subsubsection{Boost}
    1538 Like SWIG, Boost takes C++ code and generates C/C++ extension code and Python
    1539 wrappers around that extension code, which one can then use from the
    1540 Python interpreter.  \SAGE does not use Boost because it is large and
    1541 difficult to redistribute and it is meant much more for wrapping
    1542 existing C++ libraries.  Boost also uses a very complicated and
    1543 specialized build system.   
    1544 
    1545 \subsubsection{Psycho}
    1546 Psycho is a Python library that tries to guess the types of variables
    1547 in a running Python program, then generate compiled extension code
    1548 {\em on the fly} to optimize the running program.  \SAGE does not use
    1549 Psycho, because it only runs on Intel x86 machines,
    1550 %[[what about
    1551 %OS X macs, windows?]]
    1552 it has large and
    1553 unpredictable memory requirements, and so much behind-the-scenes
    1554 guessing at this level for a serious research mathematics tool is
    1555 inappropriate. 
    1556 
    1557 %[[Nonetheless: quick tutorial and examples]]
    1558 
    1559 Moreover, in my own experience with \sage, the
    1560 speedups from Psycho were not significant, except on code that had an
    1561 obvious C equivalent.  Such code can be easily rewritten in Cython,
    1562 which always results in something that is faster, more robust, and
    1563 more reliable than what Pyscho produces, and which can be debugged
    1564 using gdb.
    1565 
    1566 Though Psycho is not included with or used by \SAGE,
    1567 it is possible to try out Psycho by
    1568 installing it and putting the appropriate line at the top of a file.
    1569 
    1570 %[[put a mathy example in with timings]]
    1571 
    1572 \subsubsection{Ctypes}
    1573 Ctypes is included standard in Python 2.5, so it is clearly useful to
    1574 many Python users.  It allows one to call library
    1575 functions defined in shared object libraries directory from
    1576 interpreted Python code.  For the core of \SAGE
    1577 we have not found a use for this yet.
    1578 First, when implementing a basic type that needs to
    1579 be very fast, e.g., GMP integers, the entire implementation absolutely
    1580 must be compiled -- writing it as a combination of Python and C
    1581 library calls will be way too slow.  Second, it is often very painful
    1582 to convert Python data structures to C data structures and conversely,
    1583 and there are many opportunities for memory leaks and core dumps.
    1584 
    1585 %[[example.  like the ``install'' command in PARI.]]
    1586 
    1587 \section{Examples}
     1189Python is an interpreted language and has no declared data types for
     1190variables.  These features make it easy to write and debug, but Python
     1191code can sometimes be slow.  Cython code can look a lot like Python,
     1192but it gets translated into C code (often very efficient C code) and
     1193then compiled.  Thus it offers a language which is familiar to Python
     1194developers, but with the potential for much greater speed.
    15881195
    15891196There are several ways to create and build Cython code in \SAGE.
    15901197\begin{enumerate}
     
    16001207for you to use in the notebook.  Also, the output cell has
    16011208a  link to the C program that was compiled to create
    16021209the .so file.
    1603 \item Below we will denote notebook sessions as follows:
    1604 \begin{verbatim}
    1605 {{{
    1606   INPUT TEXT
    1607 ///
    1608   OUTPUT TEXT
    1609 }}}
    1610 ...
    1611 \end{verbatim}
    1612 The triple braces delimit cells.  You should {\em not}
    1613 actually type in the curly braces or triple forward
    1614 slashes.
    1615 
    16161210\end{enumerate}
    16171211
    16181212\item  Create an .spyx file and attach or load it
    16191213from the command line.
    16201214This is similar to creating a {\tt \%cython} cell
    16211215in the notebook but works completely from the command
    1622 line
     1216line (and not from the notebook)
    16231217
    1624 \item Create a .pyx file in the \SAGE library, add it to {\tt
    1625     SAGE_ROOT/devel/sage/setup.py} (note -- you have to follow the
    1626   template of how something else in there is done -- we simply haven't
    1627   had the time to properly document this process yet!), and type {\tt
    1628     sage -br} to build it.  This is how much of the core arithmetic in
    1629   the \SAGE library is implemented.
    1630 
    1631 
     1218\item Create a .pyx file in the \SAGE library and add a listing for it
     1219to the variable \code{ext_modules} in the file
     1220\code{SAGE_ROOT/devel/sage/setup.py}.  For example, the file
     1221\code{SAGE_ROOT/devel/sage/sage/graphs/chrompoly.pyx} has lines
     1222%skip
     1223\begin{verbatim}
     1224    Extension('sage.graphs.chrompoly',
     1225              ['sage/graphs/chrompoly.pyx']
     1226              ), \
     1227\end{verbatim}
     1228in \code{setup.py}.  Then type \code{sage -ba} to build \sage with the
     1229new code.
    16321230\end{enumerate}
    16331231
    16341232\subsection{Attaching or Loading {\tt .spyx} Files}
     1233
    16351234The easiest way to try out Cython without having to
    16361235learn anything about distutils, etc., is to create
    16371236a file with the extension {\tt spyx}, which stands
    1638 for ``SAGE Pyrex''.   Try it now!
     1237for ``SAGE Pyrex'':
    16391238
    1640 \subsection{Using the Command Line and a File}
    16411239\begin{enumerate}
    1642 \item Create a file {\tt hello.spyx}.
     1240\item Create a file \code{power2.spyx}.
    16431241\item Put the following in it:
    16441242\begin{verbatim}
    1645 def hello(name):
    1646     """
    1647     Print hello with the given name.
    1648     """
    1649     print("Hello %s"%name)
     1243def is2pow(n):
     1244    while n != 0 and n%2 == 0:
     1245        n = n >> 1
     1246    return n == 1
    16501247\end{verbatim}
    16511248\item Start the \SAGE command-line interpreter and load
    16521249the spyx file (this will fail if you do not have a C compiler
    16531250installed).
    16541251%skip
    16551252\begin{verbatim}
    1656 sage: load "hello.spyx"
    1657 Compiling hello.spyx...
    1658 sage: hello('World')
    1659 Hello World
     1253sage: load "power2.spyx"
     1254Compiling power2.spyx...
     1255sage: is2pow(12)
     1256False
    16601257\end{verbatim}
    16611258\end{enumerate}
    1662 Note that you can change {\tt hello.spyx}, then load
     1259Note that you can change \code{power2.spyx}, then load
    16631260it again and it will be recompiled on the fly.
    1664 You can also attach {\tt hello.spyx} so it is reloaded
    1665 whenever you make changes.
     1261You can also attach \code{power2.spyx} so it is reloaded
     1262whenever you make changes:
    16661263%skip
    16671264\begin{verbatim}
    1668 sage: attach "hello.spyx"
     1265sage: attach "power2.spyx"
    16691266\end{verbatim}
    1670 
    1671 \subsection{An Optimization Example}
    1672 Imagine you are writing a program that frequently computes {\tt a*b + c}
    1673 for $a,b,c$ \SAGE integers.  For example, you want to
    1674 do the following computation (timings on a Macbook Pro 2Ghz
    1675 under OS X):
     1267Cython is used for its speed.  Here is a timed test on a 2.4 GHz iMac:
    16761268%skip
    16771269\begin{verbatim}
    1678 sage: a = 939393; b = 19393; c = 39390283
    1679 sage: time for _ in range(10000): a = a*b + c           
    1680 CPU times: user 0.33 s, sys: 0.21 s, total: 0.53 s
     1270sage: time [n for n in range(10^5) if is2pow(n)]
     1271[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
     1272CPU time: 0.06 s,  Wall time: 0.06 s
    16811273\end{verbatim}
    1682 
    1683 There is no ``multiply then add'' operation built into \SAGE, and
    1684 you're very frustrated because you really want this to be as fast as
    1685 possible.  There is possibly significant overhead in the computation
    1686 of {\tt a*b + c} in Python.   
    1687 
    1688 First create a new compiled ``multiply and add'' command in \SAGE.
    1689 
    1690 %[[todo: change example below to use {\tt\_\_new\_\_}.]]
    1691 \begin{verbatim}
    1692 cimport sage.rings.integer
    1693 def muladd(sage.rings.integer.Integer a,
    1694            sage.rings.integer.Integer b,
    1695            sage.rings.integer.Integer c):
    1696     """
    1697     Compute a*b + c
    1698     """
    1699     cdef sage.rings.integer.Integer d
    1700     d = sage.rings.integer.Integer()
    1701     mpz_mul(d.value, a.value, b.value)
    1702     mpz_add(d.value, d.value, c.value)
    1703     return d
    1704 \end{verbatim}
    1705 
    1706 Do this either by putting the above code in a spyx
    1707 file or pasting it into the notebook in a cell that
    1708 starts with {\tt\%cython}.
    1709 
    1710 Now we can do the above computation more quickly:
     1274Now, the code in the file \code{power2.spyx} is valid Python, and if
     1275we copy this to a file \code{powerslow.py} and load that, we get the
     1276following:
    17111277%skip
    17121278\begin{verbatim}
    1713 sage: a = 939393; b = 19393; c = 39390283
    1714 sage: time for _ in range(10000): a = muladd(a,b,c)
    1715 CPU times: user 0.11 s, sys: 0.05 s, total: 0.15 s
     1279sage: load "powerslow.py"
     1280sage: time [n for n in range(10^5) if is2pow(n)]
     1281[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
     1282CPU time: 1.80 s,  Wall time: 1.84 s
    17161283\end{verbatim}
     1284By the way, we could gain even a little more speed -- cutting it down
     1285to 0.03 s -- with the Cython version with a type declaration, by
     1286changing \code{def is2pow(n):} to \code{def is2pow(unsigned int n):}.
    17171287
    1718 To squeeze out even more performance we can put the
    1719 entire loop in a single function.
    1720 \begin{verbatim}
    1721 cimport sage.rings.integer
    1722 def muladdloop(sage.rings.integer.Integer a,
    1723                sage.rings.integer.Integer b,
    1724                sage.rings.integer.Integer c,
    1725                int n):
    1726     cdef int i
    1727     cdef mpz_t t
    1728     cdef sage.rings.integer.Integer aa
    1729     aa = sage.rings.integer.Integer(a)
    1730     mpz_init(t)
    17311288
    1732     for i from 0 <= i < n:
    1733         mpz_mul(t, aa.value, b.value)
    1734         mpz_add(aa.value, t, c.value)
    17351289
    1736     mpz_clear(t)
    1737     return aa
    1738 \end{verbatim}
     1290\section{Other Languages}
    17391291
    1740 We then have
    1741 %skip
    1742 \begin{verbatim}
    1743 sage: a = 939393; b = 19393; c = 39390283
    1744 sage: time z=muladdloop(a,b,c,10000)
    1745 CPU times: user 0.11 s, sys: 0.00 s, total: 0.11 s
    1746 \end{verbatim}
    1747 This is a little faster. %[[no system overhead -- what's going on?]]
     1292Since \sage is based on Python, it interfaces with C and C++, as well
     1293as other languages.  See the Python documentation at
     1294\url{http://www.python.org/doc/} for more details; in particular,
     1295``Extending and Embedding the Python Interpreter'', available at
     1296\url{http://docs.python.org/ext/ext.html}, describes how to write C or
     1297C++ modules for use in Python.
    17481298
    1749 \subsection{A simple loop example: Sum of squares}
    1750 
    1751 The following \SAGE worksheet illustrates the above
    1752 sums of squares examples in the \SAGE Notebook.
    1753 
    1754 Note, we use the notation:
    1755 
    1756 %skip
    1757 \begin{verbatim}
    1758 {{{
    1759   INPUT TEXT
    1760 ///
    1761   OUTPUT TEXT
    1762 }}}
    1763 \end{verbatim}
    1764 to denote cells in the notebook.
    1765 
    1766 %[[todo put a check to avoid overflow below.  maybe switch
    1767 %to using long long, or also use that...]]
    1768 \begin{verbatim}
    1769 {{{
    1770 two = int(2)
    1771 def sumsquarespy(n):
    1772     return sum(i**two for i in xrange(1,n+1))
    1773 }}}
    1774 {{{
    1775 time v=[sumsquarespy(100) for _ in xrange(10000)]
    1776 ///
    1777 #> CPU time: 0.49 s,  Wall time: 0.49 s
    1778 }}}
    1779 {{{
    1780 %cython
    1781 # Next in cython.
    1782 def sumsquares(int n):
    1783     cdef int i, j
    1784     j = 0
    1785     for i from 1 <= i <= n:
    1786         j = j + i*i
    1787     return j
    1788 }}}
    1789 {{{
    1790 time v=[sumsquares(100) for _ in xrange(10000)]
    1791 ///
    1792 CPU time: 0.06 s,  Wall time: 0.06 s
    1793 }}}
    1794 \end{verbatim}
    1795 
    1796 \subsection{Example: 2 Power}
    1797 The following \SAGE notebook session illustrates how to implement
    1798 in Cython (entirely using the notebook) a function to quickly tell
    1799 if a C integer is a power of $2$.  The resulting function is over 200
    1800 times faster in Cython than in Python, though the actual code
    1801 is {\em identical}.   
    1802 \begin{verbatim}
    1803 {{{
    1804 %cython
    1805 def is2pow(unsigned int n):
    1806     while n != 0 and n%2 == 0:
    1807         n = n >> 1
    1808     return n == 1
    1809 }}}
    1810 \end{verbatim}
    1811 \begin{verbatim}
    1812 {{{
    1813 time [n for n in range(10^5) if is2pow(n)]
    1814 ///
    1815 [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
    1816 CPU time: 0.05 s,  Wall time: 0.06 s
    1817 }}}
    1818 \end{verbatim}
    1819 \begin{verbatim}
    1820 {{{
    1821 # The same program but in Python:
    1822 def is2pow(n):
    1823     while n != 0 and n%2 == 0:
    1824         n = n >> 1
    1825     return n == 1
    1826 }}}
    1827 \end{verbatim}
    1828 \begin{verbatim}
    1829 {{{
    1830 time [n for n in range(10^5) if is2pow(n)]
    1831 ///
    1832 [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
    1833 CPU time: 13.04 s,  Wall time: 15.08 s
    1834 }}}
    1835 \end{verbatim}
    1836 \begin{verbatim}
    1837 {{{
    1838 13.04/0.05
    1839 ///
    1840 260.79999999999995
    1841 }}}
    1842 \end{verbatim}
    1843 
    1844 %\subsection{Adding to \SAGE}
    1845 
    1846 %\subsection{Your Own {\tt setup.py}}
    1847 
    1848 \section{Cython Manual}
    1849 
    1850 \subsection{Memory Usage}
    1851 Here is the memory usage of an integer mod int object on a 32-bit system:
    1852 \begin{verbatim}
    1853 4 an int (the underlying value)
    1854 4 pointer to the parent
    1855 4 pointer to the modulus
    1856 4 pointer to the function call table
    1857 8 PyObject_HEAD  (pointer to type object and reference count)
    1858 \end{verbatim}
    1859 So a single integer mod takes 24 bytes.   
    1860 
    1861 {\em IDEA: For a small field, one could store every element, then just have
    1862 lots of copies of pointers to the same thing, which would save a ton
    1863 of memory. }
    1864 
    1865 \subsection{Forward-declaring extension types}
    1866 Extension types can be forward declared, which is necessary, e.g.,
    1867 if you have two extension types that need to refer to each other.
    1868 For example,
    1869 \begin{verbatim}
    1870 cdef class MyRing # forward declaration
    1871 
    1872 cdef class MyElement:
    1873      cdef MyRing parent
    1874 
    1875 cdef class MyRing:
    1876      cdef MyElement basis
    1877 \end{verbatim}
    1878 
    1879 If you forward declare an extension type that has a base class, you
    1880 must specify the base class in both the forward declaration and its
    1881 subsequent definition.  For example,
    1882 
    1883 \begin{verbatim}
    1884     cdef class A(B)
    1885 
    1886     ...
    1887 
    1888     cdef class A(B):
    1889         # attributes and methods
    1890 \end{verbatim}
    1891 
    1892 \subsection{Using cdef Module-Level Functions in Other Modules}
    1893 Suppose there is a cdef function in a Cython file \code{A.pyx} that is
    1894 not a method of a class, but you would like to use it from another
    1895 Cython module.  There are (at least) two options. 
    1896 One option is to make the functions methods of a cdef'd
    1897 class, then instantiate an object of that type in the other file
    1898 (this has no weird linking issues).
    1899 Another is to use ``public extern'' and link in the whole module, which we describe
    1900 below.
    1901 
    1902 For example, the file \code{A.pyx} might contain
    1903 \begin{verbatim}
    1904 cdef int sqr_int(int x):
    1905     return x*x
    1906 \end{verbatim}
    1907 You would like to call this C-level method directly from a different Cython module. 
    1908 To do so, {\em it is crucial} to declare the method public instead:
    1909 \begin{verbatim}
    1910 cdef public int sqr_int(int x):
    1911     return x*x
    1912 \end{verbatim}
    1913 Then create a file \code{A.pxd} that contains
    1914 \begin{verbatim}
    1915 cdef extern int sqr_int(int x)
    1916 \end{verbatim}
    1917 Finally, to use \code{sqr_int} in another Cython module \code{B.pyx},
    1918 do the following:
    1919 \begin{verbatim}
    1920 cimport sage.rings.A     # this is just an example absolute path
    1921 def myfunction():
    1922     print sage.rings.A.sqr_int(5)
    1923 \end{verbatim}
    1924 Any time you access one Cython module from another one,
    1925 it's necessary that the compiler know to link in the other Cython
    1926 module (\code{A.pyx} in our example).  Tell it to link
    1927 in \code{A.pyx} by including \code{A.pyx}
    1928 in the corresponding entry in \code{setup.py}.  That
    1929 entry should look something like this:
    1930 \begin{verbatim}
    1931     Extension('sage.modular.B',
    1932               ['sage/modular/B.pyx', 'sage/rings/A.pyx'])
    1933 \end{verbatim}
    1934 \note{Linking as above means an entire copy of all the code in \code{A.pyx}
    1935 is compiled into the module for \code{B.pyx}, so \code{B.so} will be larger,
    1936 and the C-level function you're accessing will {\em not} have access
    1937 to any module-scope variables in \code{A.pyx}.  This can lead to weird
    1938 core dumps, depending on the nature of your function.}
    1939 
    1940 Cython will generate a file \code{A.pxi} that also declares foo.  You
    1941 can ignore that file. 
    1942 
    1943 {\bf NOTE:} Alternatively, you could put \code{include "path/to/A.pxi"}
    1944 in \code{B.pyx} and and directly use \code{sqr_int} in \code{B.pyx} with
    1945 no scoping:
    1946 \begin{verbatim}
    1947 def myfunction():
    1948     print sqr_int(5)
    1949 \end{verbatim}
    1950 This is not recommended because it is confusing and
    1951 unpythonic to not have the scoping information.
    1952 
    1953 
    1954 \subsection{Making it possible to create weak references to extension classes}
    1955 By default, extension types do not support having weak references made
    1956 to them. You can enable weak referencing by declaring a C attribute of
    1957 type object called \code{__weakref__}. For example,
    1958 \begin{verbatim}
    1959 cdef class MyRing:
    1960    """
    1961    There is supposed to be at most one instance of this ring, so
    1962    we keep a weak reference pointer to it, so it will be deleted
    1963    from memory when there are no more user references to it.
    1964    """
    1965     cdef object __weakref__
    1966 \end{verbatim}
    1967 
    1968 \subsection{Public and Private Variables}
    1969 To  expose a variable to external Python, use \code{cdef public}, e.g.,
    1970 \begin{verbatim}
    1971 cdef class MyRing:
    1972     cdef public Integer characteristic
    1973 \end{verbatim}
    1974 Use readonly if you want to enable only readonly access.
    1975 \begin{verbatim}
    1976 cdef class MyRing:
    1977     cdef readonly Integer characteristic
    1978 \end{verbatim}
    1979 
    1980 \subsection{Arbitrary Attributes}
    1981 Reminder: Like builtin types, Cython extension types don't have a
    1982 \code{__dict__} field, so you cannot assign arbitrary attributes to
    1983 them.    That said, it is just a few lines to implement attribute
    1984 assignment, as illustrated by the following notebook session:
    1985 \begin{verbatim}
    1986 {{{
    1987 %cython
    1988 cdef class X:
    1989     cdef object __dict__
    1990     def __init__(self):
    1991         self.__dict__ = {}
    1992     def __setattr__(self, attr, val):
    1993         self.__dict__[attr] = val
    1994     def __getattr__(self, attr):
    1995         try:
    1996             return self.__dict__[attr]
    1997         except KeyError:
    1998             raise AttributeError, "object has no attribute '%s'"%attr
    1999 ///
    2000 }}}
    2001 {{{
    2002 x = X()
    2003 ///
    2004 }}}
    2005 {{{
    2006 x.a = 5
    2007 ///
    2008 }}}
    2009 {{{
    2010 x.a
    2011 ///
    2012 5
    2013 }}}
    2014 {{{
    2015 x.b
    2016 ///
    2017 Traceback (most recent call last):    x.b
    2018   File "/Volumes/HOME/sage-stable/local/lib/python2.5/", line 1, in <module>
    2019    
    2020   File "/Volumes/HOME/.sage//spyx/sage20/sage20_0.pyx", line 14, in sage20_0.X.__getattr__
    2021     raise AttributeError, "object has no attribute '%s'"%attr
    2022 AttributeError: object has no attribute 'b'
    2023 }}}
    2024 \end{verbatim}
    2025 
    2026 \subsection{Special Methods}
    2027 Certain special methods {\em must} be defined in every extension class
    2028 in order to work.  In particular, they do not derive from higher classes.
    2029 This can be very confusing if you are not aware of it.  For
    2030 example, suppose you define two classes $A$ and $B$ as follows:
    2031 \begin{verbatim}
    2032 cdef class A:
    2033    def __hash__(self):
    2034       return 0
    2035 
    2036 cdef class B(A):
    2037    def __repr__(self):
    2038       return "I am class B"
    2039 \end{verbatim}
    2040 Then $B$ will {\em not} inherit the \code{__hash__} method
    2041 from class $A$.  If you want $B$ to be hashable, you {\em must}
    2042 define a hash method explicitly for $B$, e.g.,
    2043 \begin{verbatim}
    2044 cdef class B(A):
    2045    def __hash__(self):
    2046       return A.__hash__(B)
    2047    def __repr__(self):
    2048       return "I am class B"
    2049 \end{verbatim}
    2050 
    2051 I do not know exactly which special methods do not get inherited.
    2052 Definitely \code{__hash__} and \code{__richcmp__} do not.
    2053 
    2054 
    2055 
    2056 %\section{Gotchas}
    2057 
    2058 %\subsection{How C extension classes different from Python classes}
    2059 
    2060 %\subsection{When to use {\tt cdef class} versuses {\tt def class}
    2061 %in Cython}
    2062 %def allows multiple inheritance, but no c-level data structures.
    2063 %Maybe creation is faster.   What exactly is the deal here?!
    2064 
    2065 %\subsection{\SAGE's Pyrex is Patched}
    2066 
    2067 %[[Record exactly how and why here.  Post this to the
    2068 %mailing list every once in a while. ]]
    2069 
    2070 
    2071 %\subsection{There is no +=}
    2072 %[[solution make a C macro.  need a sage.pxi that has all special
    2073 %convenience code like this in it, and is automatically included.]]
    2074 
    2075 %\subsection{Notation for C for Loops}
    2076 
    2077 %\subsection{Typos in Function Names and Variables}
    2078 
    2079 \subsection{Doctests}
    2080 If \code{x.pyx} is a Cython file, then you can do
    2081 \begin{verbatim}
    2082    sage -t x.pyx
    2083 \end{verbatim}
    2084 to test all the examples in the documentation strings
    2085 in \code{x.pyx}. 
    2086 
    2087 \subsection{Signal Handling}
    2088 Suppose you have a Cython function like
    2089 \begin{verbatim}
    2090 def foo():
    2091     cdef int n
    2092     # Some long running code that gets compiled to pure C, e.g.,
    2093     while True:
    2094         n = n + 1
    2095 
    2096 \end{verbatim}
    2097 If you call foo from the \SAGE interpreter, then when \SAGE
    2098 gets to the ``chunk of long-running C code'' pressing control-C
    2099 will not interrupt the calculation.  The only way to interrupt
    2100 the calculation is to completely kill your running copy of
    2101 \SAGE.  What a wasteful pain---surely there is a better way!
    2102 
    2103 Indeed, if you simply rewrite the above code as follows, then
    2104 suddenly control-C will work fine!
    2105 \begin{verbatim}
    2106 def foo():
    2107     cdef int n
    2108     _sig_on
    2109     while True:
    2110         n = n + 1
    2111     _sig_off
    2112 \end{verbatim}
    2113 \note{In a .pyx file which will be part of the \SAGE library,
    2114 you have to explicitly put something like
    2115 \code{include "../ext/interrupt.pxi"} in your file,
    2116 where you give in quotes the relative directory
    2117 to \code{SAGE_ROOT/devel/sage/sage/ext/interrupt.pxi}.}
    2118 
    2119 In addition to SIGINT, any code wrapped in \code{_sig_on} and
    2120 \code{_sig_off} also traps SIGABRT, SIGALRM, SIGSEV, and SIGFPE.
    2121 
    2122 {\bf IMPORTANT NOTES:}
    2123 \begin{enumerate}
    2124      \item These {\em must} always come in pairs. E.g., if you have just
    2125           a \code{_sig_off} without a corresponding \code{_sig_on}, then control-C
    2126           later in the interpreter will segfault!
    2127 
    2128         \item Do {\em not} put these in the \code{__init__} method of
    2129           a Cython extension class, or you'll get crashes.
    2130 
    2131         \item Do not put these around any Cython code that calls into
    2132           the Python API (i.e., the generated C code should have no
    2133           calls to the Python library)!!
    2134 
    2135       \item If you need to do a check of control-C inside a loop, e.g.,
    2136           when constructing a matrix, put \code{_sig_check} instead of using
    2137           \code{_sig_on} then \code{_sig_off}.
    2138 \end{enumerate}
    2139 
    2140 \subsection{Introspection}
    2141 Users currently can't view the source code of functions
    2142 from the command line.  This must change.  Ideas for fixing
    2143 this problem.
    2144 
    2145 If we can find the module name where foo is defined,
    2146 then check if the appropriate foo.pyx file is available.
    2147 Load it.  Find the function.  Figure out which if it
    2148 is defined twice.  Show all if we can't figure it out.
    2149 
    2150 If totally screwed and we can't even find the file, then
    2151 modify pyrex so it embeds extra info in the docstring.
    2152 Strip that out on displaying.
    2153 
    2154 %\section{Automatic Coercion to and From C Data Types}
    2155 %Integer literals
    2156 %type checking
    2157 %Etc.
    2158 %How to do explicit casts
    2159 
    2160 \subsection{Faster Iteration through a Python List or Tuple}
    2161 For example, define a function \code{mysum} in Cython as
    2162 follows.  Note the trick of void
    2163 pointers to avoid Cython reference counting, etc:
    2164 
    2165 \begin{verbatim}
    2166 def mysum(x):
    2167     cdef object seq
    2168     cdef int len,i
    2169     cdef object item
    2170     cdef object total
    2171     total = 0
    2172     cdef PyObject** X
    2173     X = FAST_SEQ_UNSAFE(x)
    2174     for i from 0 <=i<len:
    2175         item = <object>X[i]
    2176         total = total + item
    2177     return (total)
    2178 \end{verbatim}
    2179 The call to \code{FAST_SEQ_UNSAFE} gets direct very very
    2180 unsafe access to the underlying array of PyObject's.
    2181 
    2182 Somewhat surprisingly, this is actually {\em faster}
    2183 than Python's built in sum function:
    2184 %skip
    2185 \begin{verbatim}
    2186 sage: time sum(range(5*10^6))
    2187 CPU times: user 1.59 s, sys: 0.08 s, total: 1.67 s
    2188 12499997500000L
    2189 sage: time mysum(range(5*10^6))
    2190 CPU times: user 1.46 s, sys: 0.03 s, total: 1.49 s
    2191 12499997500000L
    2192 \end{verbatim}
    2193 
    2194 
    2195 In the next example, we illustrate the above idea by defining
    2196 several simplified versions of the \code{range} command.
    2197 
    2198 \begin{verbatim}%skip
    2199 %cython
    2200 
    2201 def f0(int n):
    2202   return eval('[%s for _ in xrange(%s)]'%(n, n))
    2203 
    2204 def f1(int n):
    2205   v = []
    2206   for i from 0 <= i < n:
    2207       v.append(i)
    2208   return v
    2209 
    2210 def f2(int n):
    2211   cdef int i
    2212   v = [None] * n
    2213  
    2214   for i from 0 <= i < n:
    2215       v[i] = i
    2216   return v
    2217 
    2218 def f3(int n):
    2219   cdef int i
    2220   w = [None] * n
    2221   cdef void** X
    2222   X = FAST_SEQ_UNSAFE(w)
    2223   for i from 0 <= i < n:
    2224       Py_DECREF(<PyObject*>X[i])
    2225       X[i] = PyInt_FromLong(i)
    2226   return w
    2227 
    2228 def f4(int n):
    2229   cdef int i
    2230   v = PyList_New(n)
    2231   for i from 0 <= i < n:
    2232       PyList_SET_ITEM(v, i, <object>PyInt_FromLong(i))
    2233       #WARNING -- maybe I need to incref the ref to the long?!
    2234   return v
    2235 \end{verbatim}
    2236 
    2237 The fourth one is the fastest.
    2238 
    2239 \subsection{getslice}
    2240 For fastest speed, use this declaration:
    2241 \begin{verbatim}
    2242 def __getslice__(self, Py_ssize_t i, Py_ssize_t j):
    2243     ...
    2244 \end{verbatim}
    2245 
    2246 %\section{The Python/C API}
    2247 % The {\tt object} type.
    2248 %\section{C Header Files}
    2249 %\section{Writing Your Code in a Mix of Cython and C/C++}
    2250 %\section{Splitting Implementations Across Multiple Files}
    2251 %\section{Wrapping C++ Code}
    2252 %Cython code compiles with C++ compiler.  Using pyrexembed.
    2253 %\section{Other Optimization Tricks}
    2254 
    2255 %\section{Misc}
    2256 %\subsection{Class methods}
    2257 %In a cdef'd class this is not possible at all.
    2258 
    2259 \subsection{List Comprehensions and In-Place operators}
    2260 List comprehensions and in-place operators are fully supported in Cython.
    2261 (They are not supported in Pyrex, but Robert Bradshaw added support
    2262 for them to Cython.)
    2263 
    2264 \subsection{Cython: Dealing with weird build errors}
    2265 \begin{itemize}
    2266 \item {\bf Problem:} You get an error like this
    2267 \begin{verbatim}%skip
    2268 ...
    2269 lude/python2.5 -c sage/modules/complex_double_vector.c
    2270       -o build/temp.macosx-10.3-i386-2.5/sage/modules/complex_double_vector.o -w
    2271 sage/modules/complex_double_vector.c:366:
    2272        error: field '__pyx_base' has incomplete type
    2273 sage/modules/complex_double_vector.c:373:
    2274        error: field '__pyx_base' has incomplete type
    2275 error: command 'gcc' failed with exit status 1
    2276 sage: There was an error installing modified sage library code.
    2277 \end{verbatim}
    2278 {\bf Solution:} In this case I added \code{cimport sage.structure.element} to \code{complex_double_vector.pxd}
    2279 and the problem disappeared.  In general, look at the generated C file to see what type is incomplete (i.e.,
    2280 not properly declared), then add an explicit cimport to your pxd file (or top of your pyx file if you
    2281 don't have a pxd file).
    2282 
    2283 \item {\bf Problem:} Errors such as "XXX does not appear to be the correct type object".\\
    2284 {\bf Solution:} This can happen when the build system doesn't properly detect a dependency
    2285 and regenerate the corresponding files.  If you know which .pyx file should be recompiled,
    2286 just touch it.  If you don't, you can always type \code{sage -ba} to brutally
    2287 rebuild everything, which will take about 3-5 minutes.
    2288 
    2289 \end{itemize}
    2290 
    2291 %\section{To Add}
    2292 %\begin{verbatim}
    2293 %Parsifal Herzog wrote:
    2294 % >     cdef va_list ap   <--- line 112
    2295 % >     cdef voidptr oi
    2296 % >     va_start(ap, o1)
    2297 % >     oi = va_arg(ap, voidptr)
    2298 % >     while oi != 0:
    2299 % >         l.append(<object>oi)
    2300 % >         oi = va_arg(ap, voidptr)
    2301 % >     va_end(ap)
    2302  
    2303 % Here is an example how to use va_args:
    2304  
    2305 % http://codespeak.net/svn/lxml/trunk/src/lxml/cstd.pxd
    2306  
    2307 % The macros are defined at the the top of this file:
    2308  
    2309 % http://codespeak.net/svn/lxml/trunk/src/lxml/etree_defs.h
    2310  
    2311 % Stefan
    2312  
    2313  
    2314 % _______________________________________________
    2315 % Pyrex mailing list
    2316 % Pyrex@lists.copyleft.no
    2317 % http://lists.copyleft.no/mailman/listinfo/pyrex
    2318 % \end{verbatim}
    2319 
    2320 % \begin{verbatim}
    2321 % > Ok
    2322 % >
    2323 % > about my casting question. I should have declared
    2324 % >
    2325 % > cdef int c_length
    2326 % >
    2327 % > in which case the code works correctly. However in one place I had
    2328 % > forgot the int and just typed
    2329 % >
    2330 % > cdef c_length
    2331 % >
    2332 % > and this led to the example I sent. What type of object is created if 
    2333 % > you declare with cdef but forget the type.
    2334 
    2335 % The default is a Python object.  It's equivalent to
    2336 %     cdef object c_length.
    2337 
    2338 % William
    2339 % \end{verbatim}
    2340 
    2341 \section{C++ Nuances}
    2342 (This section was written by Joel B. Mohler.  Many of the tricks come from
    2343 Martin Albrecht and David Harvey.)
    2344 
    2345 Cython can generate C++ for wrapping of C++ libraries.  There are just a
    2346 few things to keep in mind though.  Cython doesn't know specifically about
    2347 C++ features like function and operator overloading.  Therefore, your library
    2348 must have unique function names or you must alias them.  Other issues
    2349 which arise have to do with functions taking references and pointer
    2350 dereferencing since C does not support either of these.
    2351 
    2352 You can see real live examples for many of these concepts in the NTL
    2353 wrapper classes.  These are in the \SAGE tree at \code{sage/libs/ntl/}.
    2354 
    2355 \subsection{Aliases for Overloading (or other purposes)}
    2356 
    2357 Cython doesn't support overloaded functions, but C++ libraries readily use
    2358 this feature.  Cython does provide a very simple way to alias these functions
    2359 which we illustrate with an example borrowed from the NTL wrapper code.
    2360 
    2361 \begin{verbatim}
    2362 cdef extern from "ntl_wrap.h":
    2363     # really, this is from NTL/ZZ.h
    2364     ctypedef struct ZZ_c "struct ZZ":
    2365         pass
    2366     void ZZ_add "add"( ZZ_c x, ZZ_c a, ZZ_c b)
    2367 
    2368     #### ZZ_p_c
    2369     ctypedef struct ZZ_p_c "struct ZZ_p":
    2370         pass
    2371     void ZZ_p_add "add"( ZZ_p_c x, ZZ_p_c a, ZZ_p_c b)
    2372 \end{verbatim}
    2373 
    2374 Notice that NTL supports the \code{add} function for all of its many types.
    2375 We rename them for use in Cython as \code{<Type>_add} where \code{<Type>}
    2376 is the NTL type name of the objects being added.
    2377 
    2378 Another type of aliasing in that example is the structure \code{ZZ} is called
    2379 \code{ZZ_c}.  We did this simply to alleviate confusion with the \SAGE integer
    2380 class commonly known as \code{ZZ}.  This trick is also useful if you have a
    2381 C++ library that uses namespaces.  (Actually NTL does use namespaces,
    2382 but we were lazy and instead inserted a \code{using namespaces NTL;} in
    2383 a header file).
    2384 
    2385 \subsection{Pointers and References}
    2386 
    2387 Given a pointer to a C/C++ type, Cython does not support the use of '*' to
    2388 dereference this pointer.  Here's a way around that:
    2389 \begin{verbatim}
    2390     cdef Foo *t
    2391     t = <something>
    2392     function_that_takes_a_reference(t[0])
    2393 \end{verbatim}
    2394 In C, \code{t[0]} is the first element of the array pointed to by \code{t}.  Remember
    2395 that C arrays are simply pointers.  Note that this trick is rarely necessary in C
    2396 since most C code operates by passing around pointers to structures.  They are
    2397 not ordinarily dereferenced.
    2398 
    2399 \subsection{Embedding C++ objects in Cython extension structures}
    2400 
    2401 Suppose that you have a C++ class (I'll call it Foo) which you want to wrap for
    2402 access from python.  There would be two basic ways to go about managing
    2403 this structure:
    2404 
    2405 \begin{itemize}
    2406 \item Initialize a point to the object in the \code{__new__} function in my
    2407 python/Cython object and delete in the  \code{__dealloc__}.  This method
    2408 is pretty easy to understand from a C++ perspective.
    2409 \item Declare the object as member of the Cython cdef'ed class.  This method
    2410 is not quite so straightforward, but it saves you a memory allocation.  This might
    2411 be a decent speed gain for elements of some ring where an object is allocated
    2412 for every arithmetic operation.
    2413 \end{itemize}
    2414 
    2415 We let the details of the first option as an exercise to the reader and provide a
    2416 more elaborate description for the second option.  Use code like the following
    2417 to embed the object in the Cython extension class structure:
    2418 \begin{verbatim}
    2419 cdef class FooWrapper:
    2420      cdef Foo x
    2421 \end{verbatim}
    2422 
    2423 This may appear to work fine because many C++ constructors don't do a whole
    2424 lot of interesting work and Cython zeroes out the memory in these structures.
    2425 However, there is a problem since neither the constructor nor destructor are called
    2426 for Foo since the memory is allocated with malloc (or PyMalloc or something).
    2427  So, you need to call the constructor and destructor manually.  The rest of this
    2428 section provides a simple way to do so.
    2429 
    2430 The main trick is that we need to call the constructor of Foo, but the memory
    2431 is already allocated.  An important nuance of the C++ \code{new} operator is
    2432 described here:
    2433 \url{http://publib.boulder.ibm.com/infocenter/macxhelp/v6v81/index.jsp?topic=/com.ibm.vacpp6m.doc/language/ref/clrc05cplr199.htm}.
    2434 There is a C++ header file in the \code{sage_c_lib} which defines some templated
    2435 functions to make writing wrappers for C++ classes more convenient.  You can
    2436 make the following declarations in a pxd or pxi file for use from Cython.
    2437 \begin{verbatim}
    2438 cdef extern from "Foo.h":
    2439     ctypedef struct Foo:
    2440         pass
    2441 
    2442 cdef extern from "ccobject.h":
    2443     # Some non-allocating versions
    2444     Foo* Foo_construct "Construct<Foo>"(void *mem)
    2445     void Foo_destruct "Destruct<Foo>"(Foo *mem)
    2446 
    2447     # some allocating versions for completeness
    2448     Foo* Foo_new "New<Foo>"()
    2449     void Foo_delete "Delete<Foo>"(Foo *mem)
    2450 \end{verbatim}
    2451 
    2452 Now, we need to call these functions we defined from our Cython
    2453 \code{__new__} and \code{__dealloc__} functions.
    2454 \begin{verbatim}
    2455 cdef class FooWrapper:
    2456     def __init__(self, ...):
    2457         pass
    2458 
    2459     # Note that the parameters to __new__ must be identical to __init__
    2460     # This is due to some Cython vagary
    2461     def __new__(self, ...):
    2462         Foo_construct(&self.x)
    2463 
    2464     def __dealloc__(self):
    2465         Foo_destruct(&self.x)
    2466 \end{verbatim}
    2467 
    2468 \chapter{\SAGE{} Interfaces}
    2469 This chapter is about SAGE interfaces, both
    2470 the C/C++ level interfaces, and the pseduo-tty interfaces,
    2471 in \SAGE programming.
    24721299
    24731300\section{The PARI C-library Interface}
     1301
    24741302(This chapter was written by Martin Albrecht.)
    24751303
    24761304Here is the step-by-step guide to adding a new PARI functions to \SAGE.
    24771305We use the Frobenius form of a matrix as an example.
    24781306
    24791307The heavy lifting for matrices over integers is implemented using the
    2480 PARI library. To compute the frobenius form in PARI the
     1308PARI library. To compute the Frobenius form in PARI the
    24811309\code{matfrobenius} function is used.
    24821310
    24831311There are two ways to interact with the PARI library from \SAGE: The gp
    2484 interface uses the gp interpreter and the PARI interface uses direct calls to
    2485 the PARI C functions and is the preferred way as it is much faster.
     1312interface uses the gp interpreter, and the PARI interface uses direct calls to
     1313the PARI C functions --- this is the preferred way as it is much
     1314faster.  Thus this section focuses on using PARI.
    24861315
    2487 So we need to add a new method to the gen class which is the abstract
    2488 representation of all PARI library objects. That means that if we add a
    2489 method to this class every PARI object regardless if it is a number,
    2490 polynomial or matrix will have our new method. So you can do
    2491 \code{pari(1).matfrobenius()} but you will receive a PariError in this case.
     1316So we will add a new method to the gen class: this is the abstract
     1317representation of all PARI library objects. That means that once we add a
     1318method to this class, every PARI object, whether it is a number,
     1319polynomial or matrix, will have our new method. So you can do
     1320\code{pari(1).matfrobenius()}, but since PARI wants to apply
     1321\code{matfrobenius} to matrices, not numbers, you will receive a
     1322PariError in this case.
    24921323
    2493 The gen class is defined in sage/libs/pari/gen.pyx where we add the method
    2494 \code{matfrobenius}:
     1324The gen class is defined in
     1325\code{SAGE_ROOT/devel/sage/sage/libs/pari/gen.pyx}, and this is where we add
     1326the method \code{matfrobenius}:
    24951327
    24961328\begin{verbatim}
    24971329
     
    25071339        return self.new_gen(matfrobenius(self.g, flag))
    25081340\end{verbatim}
    25091341
    2510 The \verb+_sig_on+ statement is some magic to prevent SIGSEGVs from the pari C
     1342The \code{_sig_on} statement is some magic to prevent SIGSEGVs from the PARI C
    25111343library to crash the \SAGE interpreter by catching these signals. The
    2512 \verb+self.new_gen()+ call constructs a new SAGE-python-gen object from a given
     1344\code{self.new_gen()} call constructs a new SAGE-python-gen object from a given
    25131345pari-C-gen where the pari-C-gen is stored as the SAGE-python-gen.g attribute.
    2514 The matfrobenius call is just a call to the pari C library function
    2515 "matfrobenius" with the appropriate parameters.
     1346The \code{matfrobenius} call is just a call to the PARI C library function
     1347\code{matfrobenius} with the appropriate parameters.
    25161348
    2517 The information which function to call and how to call it can be retrieved
    2518 from the PARI users manual (note: \SAGE includes the development version of
    2519 PARI so check the development version's user manual). Looking for
     1349The information about which function to call and how to call it can be retrieved
     1350from the PARI user's manual (note: \SAGE includes the development version of
     1351PARI, so check that version of the user's manual). Looking for
    25201352\code{matfrobenius} you can find:
    2521 \verb+"The library syntax is matfrobenius(M,flag)"+
     1353\verb+"The library syntax is matfrobenius(M,flag)"+.
    25221354
    2523 Please not that the pari C function may have a different name than gp function
    2524 (see e.g. \code{mathnf}), so always check with the manual.
     1355In case you are familiar with gp,
     1356please note that the PARI C function may have a different name than
     1357the corresponding gp function
     1358(for example, see \code{mathnf}), so always check the manual.
    25251359
    2526 At this point we are done with the PARI interface and may add some more
     1360At this point we are done with the PARI interface and can add some more
    25271361interfaces around it for convenience:
    2528 
    2529 First we add a functional representation of the method to
    2530 sage/libs/pari/functional.py so we can do:
    2531 \verb+matfrobenius(<some.pari.gen>)+ additionally to
    2532 \verb+<some.pari.gen>.matfrobenius()+.
    2533 
     1362first we add a functional representation of the method to
     1363\code{SAGE_ROOT/devel/sage/sage/libs/pari/functional.py}
     1364so we can do
     1365\code{matfrobenius(<some.pari.gen>)} as well as
     1366\code{<some.pari.gen>.matfrobenius()}:
    25341367\begin{verbatim}
    25351368def matfrobenius(self): return pari(self).matfrobenius()
    25361369\end{verbatim}
    25371370
    2538 Then we also add a frobenius(flag) method to the matrix_integer class where we
    2539 call the \code{matfrobenius()} method on the \verb+_pari_()+ object associated with the
    2540 matrix after doing some sanity checking. Then we need to convert the pari gen
    2541 to some meaningful \SAGE objects depending on the return value as described in
    2542 the PARI user's manual.
    2543 
     1371Then we also add a \code{frobenius(flag)} method to the
     1372\code{matrix_integer} class where we call the \code{matfrobenius()}
     1373method on the PARI object associated to the matrix after doing some
     1374sanity checking. Then we convert output from PARI to \SAGE objects:
    25441375\begin{verbatim}
    25451376    def frobenius(self,flag=0):
    25461377        """
    2547         Return the Frobenius form of this matrix.
     1378        If flag is 0 (the default value), return the Frobenius
     1379            form of this matrix.
    25481380        If flag is 1, return only the elementary divisors.
    2549         If flag is 2, return a two-components vector [F,B]
    2550         where F is the Frobenius form and B is the basis change
    2551         so that M=B^-1*F*B.
     1381        If flag is 2, return a two-component vector [F,B]
     1382            where F is the Frobenius form and B is the basis change
     1383            so that M=B^-1*F*B.
    25521384
    25531385        INPUT:
    25541386           flag -- 0,1 or 2 as described above
     
    25791411        elif flag==1:
    25801412            r = polynomial_ring.PolynomialRing(self.base_ring())
    25811413            #BUG: this should be handled in PolynomialRing not here
    2582             return [eval(str(x).replace("^","**"),{},r.gens_dict()) for x in v.python_list()]
     1414            return [eval(str(x).replace("^","**"),{},r.gens_dict())
     1415                    for x in v.python_list()]
    25831416        elif flag==2:
    2584             F = matrix_space.MatrixSpace(rational_field.RationalField(),self.nrows())(v[0].python())
    2585             B = matrix_space.MatrixSpace(rational_field.RationalField(),self.nrows())(v[1].python())
     1417            F = matrix_space.MatrixSpace(rational_field.RationalField(),
     1418                                         self.nrows())(v[0].python())
     1419            B = matrix_space.MatrixSpace(rational_field.RationalField(),
     1420                                         self.nrows())(v[1].python())
    25861421            return F,B
    2587 
    25881422\end{verbatim}
    25891423
    2590 Then we add some examples, wait for make test to complete without errors and
    2591 commit.
    25921424
    25931425\section{GAP}
    25941426
     
    25961428
    25971429Wrapping a GAP function in \SAGE is a matter of writing a program in
    25981430Python which uses the pexpect interface to pipe various commands to
    2599 GAP and read back the input into \SAGE.  This can range from easy to
     1431GAP and read back the input into \SAGE.  This is sometimes easy, sometimes
    26001432hard.
    26011433
    2602 For example, suppose we want to make a wrapper for computation of the
     1434For example, suppose we want to make a wrapper for the computation of the
    26031435Cartan matrix of a simple Lie algebra. The Cartan matrix of $G_2$ is
    26041436available in GAP using the commands
    26051437%skip
     
    26131445(Incidentally, most of the GAP Lie algebra implementation was written by
    26141446Thomas Breuer, Willem de Graaf and Craig Struble.)
    26151447
    2616 In \SAGE, one can simply type
     1448In \SAGE, one can access these commands by typing
    26171449
    26181450\begin{verbatim}
    26191451sage: L = gap.SimpleLieAlgebra('"G"', 2, 'Rationals'); L
     
    26261458\end{verbatim}
    26271459Note the \code{'"G"'} which is evaluated in Gap as the string \code{"G"}.
    26281460
    2629 Using this example, the purpose of this section is
    2630 to show how one might write a Pyhon/\SAGE program
    2631 who's input is, say, \code{('G',2)} and who's output
    2632 is the matrix above (but as a \SAGE Matrix - see
    2633 \code{matrix.py} in \code{sage/matrix/matrix.py}).
     1461The purpose of this section is to use this example
     1462to show how one might write a Python/\SAGE program
     1463whose input is, say, \code{('G',2)} and whose output
     1464is the matrix above (but as a \SAGE Matrix --- see
     1465the code in the directory \code{SAGE_ROOT/devel/sage/sage/matrix/} and the
     1466corresponding parts of the \sage reference manual).
    26341467
    26351468First, the input must be converted into strings consisting of legal
    26361469GAP commands. Then the GAP output, which is also a string, must be
     
    26621495
    26631496    L = gap.SimpleLieAlgebra('"%s"'%type, rank, 'Rationals')
    26641497    R = L.RootSystem()
    2665     sM  = R.CartanMatrix()
    2666     ans = eval(str(sM))
    2667     MS  = MatrixSpace(ZZ, rank)
    2668     return MS(ans)
    2669 \end{verbatim}
    2670 The output {\tt ans} is a Python list. The last two lines convert that
    2671 list to a \SAGE class object Matrix instance.
    2672 
    2673 Alternatively, one could code the body of the above function in
    2674 a more pythonic way as follows:
    2675 %skip
    2676 \begin{verbatim}
    2677     L = gap.new(SimpleLieAlgebra("%s", %s, Rationals);'%(type, rank))
    2678     R = L.RootSystem()
    26791498    sM = R.CartanMatrix()
    26801499    ans = eval(str(sM))
    26811500    MS  = MatrixSpace(QQ, rank)
    26821501    return MS(ans)
    26831502\end{verbatim}
     1503The output \code{ans} is a Python list. The last two lines convert that
     1504list to an instance of the \sage class \code{Matrix}.
     1505
     1506Alternatively, one could replace the first line of the above function
     1507with this:
     1508%skip
     1509\begin{verbatim}
     1510    L = gap.new(SimpleLieAlgebra("%s", %s, Rationals);'%(type, rank))
     1511\end{verbatim}
    26841512
    26851513Defining ``easy'' and ``hard'' is subjective, but here is one
    2686 definition: an example is ``easy'' if there is already a corresponding
    2687 class object in Python or \SAGE for the output data type of the GAP
     1514definition: wrapping a GAP function is ``easy'' if there is already a corresponding
     1515class in Python or \SAGE for the output data type of the GAP
    26881516function you are trying to wrap.  For example, wrapping any GUAVA
    26891517(GAP's error-correcting codes package) function is ``easy'' since
    26901518error-correcting codes are vector spaces over finite fields and GUAVA
     
    27071535\item[(e)]
    27081536integers.
    27091537\end{itemize}
    2710 \SAGE already has a class object for each of these.
     1538\SAGE already has classes for each of these.
    27111539
    27121540A ``hard'' example is left as an exercise!  Here are a few ideas.
    27131541
    27141542\begin{itemize}
    27151543
    2716 \item Write a wrapper for GAP's {\tt FreeLieAlgebra} function (or,
     1544\item Write a wrapper for GAP's \code{FreeLieAlgebra} function (or,
    27171545  more generally, all the finitely presented Lie algebra fuunctions in
    27181546  GAP).  This would require creating new Python objects.
    27191547
    2720 \item Write a wrapper for GAP's {\tt FreeGroup} function (or, more
     1548\item Write a wrapper for GAP's \code{FreeGroup} function (or, more
    27211549  generally, all the finitely presented groups fuunctions in GAP).
    27221550  This would require writing some new Python objects.
    27231551
     
    27281556\end{itemize}
    27291557
    27301558\section{Singular}
     1559
    27311560(The first version of this chapter was written by David Joyner.)
    27321561
    2733 Using Singular functions from \SAGE is not much different conceptually than using
    2734 GAP functions from \SAGE. This can range from very easy to hard, depending on how
    2735 much of the data structure of the output of the Singular function is already
    2736 present in \SAGE.
     1562Using Singular functions from \SAGE is not much different conceptually
     1563than using GAP functions from \SAGE. As with GAP, this can range from
     1564easy to hard, depending on how much of the data structure of the
     1565output of the Singular function is already present in \SAGE.
    27371566
    2738 First, some terminology. For us, a \emph{curve} $X$ over a finite field $F$ is a
     1567First, some terminology. For us, a \emph{curve} $X$ over a finite field $F$ is an
    27391568equation of the form $f(x,y)=0$, where $f\in F[x,y]$ is a polynomial. It may or
    27401569may not be singular.  A \emph{place of degree} $d$ is a Galois orbit of $d$
    27411570points in $X(E)$, where $E/F$ is degree $d$. For example, a place of degree $1$
     
    29511780% }
    29521781
    29531782
    2954 >From looking at the output, notice that our wrapper program will need to
    2955 parse the string represented by $L$ above.
    2956 
    2957 Let us write a separate program to do just that. This requires figuring out how
     1783From looking at the output, notice that our wrapper program will need to
     1784parse the string represented by $L$ above, so
     1785let us write a separate program to do just that. This requires figuring out how
    29581786to determine where the coordinates of the points are placed in the string L.
    29591787Python has some very useful string manipulation commands to do just that.
    29601788
     
    30121840
    30131841
    30141842Now it is an easy matter to put these ingredients together into a \SAGE function
    3015 which takes as input a triple $(f,F,d)$: finite field $F$ \emph{of prime order},
    3016 a polynomial $f$ in $F[x,y]$ defining $X:\  f(x,y)=0$ (note the variables $x,y$
    3017 must be used), and the degree $d$. The output is the number of points of places
    3018 in $X$ of degree $d=1$ over $F$. At the moment, their is no ``translation''
     1843which takes as input a triple $(f,F,d)$:
     1844a polynomial $f$ in $F[x,y]$ defining $X:\  f(x,y)=0$ (note that the variables $x,y$
     1845must be used),
     1846a finite field $F$ \emph{of prime order},
     1847and the degree $d$. The output is the number of places
     1848in $X$ of degree $d=1$ over $F$. At the moment, there is no ``translation''
    30191849between elements of $GF(p^d)$ in Singular and \SAGE unless $d=1$. So, for this
    30201850reason, we restrict ourselves to points of degree one.
    30211851
     
    30521882    return points_parser(L,F)
    30531883
    30541884\end{verbatim}
    3055 The triple quoted string documentation is important! It not only helps users
    3056 understand your function's use and syntax but also, if the function is included
    3057 in \SAGE, will be reproduced in the printed and online documentation, and
    3058 \emph{automatically tested} (even the output) before each \SAGE release.
    3059 
    3060 Note also that the ordering returned by this \SAGE function is exactly the same
     1885Note that the ordering returned by this \SAGE function is exactly the same
    30611886as the ordering in the Singular variable \code{POINTS}.
    30621887
    30631888%\comment{
     
    30761901%number_of_places(f,F,d)
    30771902%}
    30781903
    3079 Some examples:
     1904One more example (in addition to the one in the docstring):
    30801905
    30811906%skip
    30821907\begin{verbatim}
    3083 sage: F = GF(5)
    3084 sage: R = MPolynomialRing(F,2,names = ["x","y"])
    3085 sage: x,y = R.gens()
    3086 sage: f = y^2-x^9-x
    3087 sage: places_on_curve(f,F)
    3088       ((0, 1, 0), (3, 4, 1), (0, 0, 1), (2, 3, 1), (3, 1, 1), (2, 2, 1))
    3089 
    30901908sage: F = GF(2)
    30911909sage: R = MPolynomialRing(F,2,names = ["x","y"])
    30921910sage: x,y = R.gens()
     
    30951913      ((0, 1, 0), (1, 0, 0), (0, 0, 1))
    30961914\end{verbatim}
    30971915
    3098 \section{Another Approach}
    3099 There is also a more python-like interface to Singular.  Using this
     1916\section{Singular: Another Approach}
     1917
     1918There is also a more Python-like interface to Singular.  Using this
    31001919the code is much simpler, as illustrated below.   First we demonstrate
    31011920computing the places on a curve in a particular case.
    31021921
     
    31161935sage: [(L[i][1], L[i][2], L[i][3]) for i in range(1,7)]
    31171936      [(0, 1, 0), (-2, 1, 1), (0, 0, 1), (2, 2, 1), (-2, -1, 1), (2, -2, 1)]
    31181937\end{verbatim}
    3119 The \code{name} method of a Singular object returns the name
    3120 of that object in the Singular interpreter, so that it can
    3121 be used as input to a Singular function.
    31221938
    3123 Next we implement the general function (we omit the docstring, which
     1939Next we implement the general function (for brevity we omit the docstring, which
    31241940is the same as above).  Note that the \code{point_parser} function
    31251941is not required.
    31261942
     
    31491965used only the barest minimum of that interface.
    31501966
    31511967\section{Creating a new Pseudo-tty Interface}
    3152 \SAGE pseudo-tty interfaces can be created that allow \SAGE to
    3153 work with an almost completely arbitrary command line program,
    3154 and don't require any modification or extensions to the
    3155 command line program.   They are also {\em surprisingly} fast
     1968
     1969You can create \SAGE pseudo-tty interfaces that allow \SAGE to
     1970work with almost any command-line program,
     1971and which don't require any modification or extensions to that
     1972program.  They are also {\em surprisingly} fast
    31561973and flexible (given how they work!), because all IO is
    31571974buffered, and because interaction between \sage and the
    3158 the command line program can be non-blocking (asynchronous),
    3159 using the \code{_send}, \code{_so_far}, and \code{_get}
    3160 methods.
     1975command line program can be non-blocking (asynchronous); this is
     1976because they all derive from the \sage class \code{Expect}, which
     1977handles the communication between \sage and the external process.
    31611978
    3162 % To create a new interface to a computer algebra system,
    3163 % do the following:
    3164 % \begin{enumerate}
    3165 % \item Copy
     1979For example, here is part of the file
     1980\code{SAGE_ROOT/devel/sage/sage/interfaces/octave.py}, which defines
     1981an interface between \sage and Octave, an open-source program
     1982for doing numerical computations, among other things.
     1983%skip
     1984\begin{verbatim}
     1985import os
     1986from expect import Expect, ExpectElement
    31661987
    3167 % \section{Projects}
    3168 % For the interested reader, here are a few ideas for some other wrappers,
    3169 % which are not yet implemented in \SAGE.
     1988class Octave(Expect):
     1989\end{verbatim}
     1990The first two lines import the library \code{os}, which contains
     1991operating system routines, and also class \code{Expect}, which is the
     1992basic class for interfaces.  The third line defines the class
     1993\code{Octave}: it derives from \code{Expect}.  After this comes a
     1994docstring, which we omit here -- see the file for details.  Next comes:
     1995%skip
     1996\begin{verbatim}
     1997    def __init__(self, maxread=100, script_subdirectory="", logfile=None,
     1998                 server=None, server_tmpdir=None):
     1999        Expect.__init__(self,
     2000                        name = 'octave',
     2001                        prompt = '>',
     2002                        command = "octave --no-line-editing --silent",
     2003                        maxread = maxread,
     2004                        server = server,
     2005                        server_tmpdir = server_tmpdir,
     2006                        script_subdirectory = script_subdirectory,
     2007                        restart_on_ctrlc = False,
     2008                        verbose_start = False,
     2009                        logfile = logfile,
     2010                        eval_using_file_cutoff=100)
     2011\end{verbatim}
     2012This uses the class \code{Expect} to set up the Octave interface.
     2013%skip
     2014\begin{verbatim}
     2015    def set(self, var, value):
     2016        """
     2017        Set the variable var to the given value.
     2018        """
     2019        cmd = '%s=%s;'%(var,value)       
     2020        out = self.eval(cmd)
     2021        if out.find("error") != -1:
     2022            raise TypeError, "Error executing code in Octave\nCODE:\n\t%s\nOctave ERROR:\n\t%s"%(cmd, out)
    31702023
    3171 % \begin{itemize}
     2024    def get(self, var):
     2025        """
     2026        Get the value of the variable var.
     2027        """
     2028        s = self.eval('%s'%var)
     2029        i = s.find('=')
     2030        return s[i+1:]
    31722031
    3173 % \item
    3174 % Write a \SAGE function, possibly containing Singular code,
    3175 % which ``translates'' the Singular generator
    3176 % \code{a} of $\GF(p^d)$ to a \SAGE generator of $\GF(p^d)$.
     2032    def console(self):
     2033        octave_console()
     2034\end{verbatim}
     2035These let users type \code{octave.set('x', 3)}, after which
     2036\code{octave.get('x')} returns \code{' 3'}.  Running
     2037\code{octave.console()} dumps the user into Octave interactive shell.
     2038%skip
     2039\begin{verbatim}
     2040    def solve_linear_system(self, A, b):
     2041        """
     2042        Use octave to compute a solution x to A*x = b, as a list.
    31772043
    3178 % \item
    3179 % Write a wrapper for Singular's
    3180 % \code{BrillNoether} function to compute bases of Riemann-Roch spaces.
    3181 % This would not require writing some new Python objects.
     2044        INPUT:
     2045            A -- mxn matrix A with entries in QQ or RR
     2046            b -- m-vector b entries in QQ or RR (resp)
    31822047
    3183 % \item
    3184 % Write a wrapper for Singular's
    3185 % \code{groebner} function to compute Groebner bases.
    3186 % This would not require writing some new Python objects.
     2048        OUTPUT:
     2049            An list x (if it exists) which solves M*x = b
    31872050
    3188 % \item
    3189 % Write a wrapper for Singular's method for
    3190 % ``Solving systems of polynomial equations'' in
    3191 % \url{http://www.singular.uni-kl.de/Manual/3-0-0/sing_502.htm\#SEC554}
     2051        EXAMPLES:
     2052            sage: M33 = MatrixSpace(QQ,3,3)
     2053            sage: A   = M33([1,2,3,4,5,6,7,8,0])
     2054            sage: V3  = VectorSpace(QQ,3)
     2055            sage: b   = V3([1,2,3])
     2056            sage: octave.solve_linear_system(A,b)    # requires optional octave
     2057            [-0.33333299999999999, 0.66666700000000001, -3.5236600000000002e-18]
    31922058
    3193 % \item
    3194 % Write a wrapper for Singular's method for constructing and decoding
    3195 % ``AG codes'' in
    3196 % \url{http://www.singular.uni-kl.de/Manual/3-0-0/sing_503.htm\#SEC555}
    3197 % \end{itemize}
     2059        AUTHOR: David Joyner and William Stein
     2060        """
     2061        m = A.nrows()
     2062        n = A.ncols()
     2063        if m != len(b):
     2064            raise ValueError, "dimensions of A and b must be compatible"
     2065        from sage.matrix.all import MatrixSpace
     2066        from sage.rings.all import QQ
     2067        MS = MatrixSpace(QQ,m,1)
     2068        b  = MS(list(b)) # converted b to a "column vector"
     2069        sA = self.sage2octave_matrix_string(A)
     2070        sb = self.sage2octave_matrix_string(b)
     2071        self.eval("a = " + sA )
     2072        self.eval("b = " + sb )
     2073        soln = octave.eval("c = a \\ b")
     2074        soln = soln.replace("\n\n ","[")
     2075        soln = soln.replace("\n\n","]")
     2076        soln = soln.replace("\n",",")
     2077        sol  = soln[3:]
     2078        return eval(sol)
     2079\end{verbatim}
     2080This code defines the method \code{solve_linear_system}, which works
     2081as documented.
    31982082
     2083These are only excerpts from \code{octave.py}; check that file for
     2084more definitions and examples.  Look at other files in the directory
     2085\code{SAGE_ROOT/devel/sage/sage/interfaces/} for examples of
     2086interfaces to other software packages.
    31992087
    3200 \chapter{Mercurial: The \sage Source Control System}
     2088
     2089
     2090\chapter{The \SAGE Manuals}
     2091\label{ch:manuals}
     2092
     2093This chapter describes how to modify the \SAGE manuals.
     2094\sage's manuals are written in \Latex, and to edit them, you just need
     2095to edit the appropriate file. 
     2096
     2097If, for example, you want to change the \sage tutorial, then you
     2098should start by modifying the file
     2099\code{SAGE_ROOT/devel/doc/tut/tut.tex}.  Then to build a dvi or pdf file
     2100with your changes, type \code{./build_dvi} or
     2101\code{./build_pdf} in the \code{devel/doc/tut} directory.  You'll get
     2102a file \code{tut.dvi} or \code{tut.pdf}, which you should inspect.
     2103You should also run \code{sage -t tut.tex} to test all of the examples
     2104-- see Section~\ref{ch:testing}.
     2105You can build an HTML
     2106version of the tutorial by typing \code{make tut} in the
     2107\code{devel/doc} directory.  Once you've done this, you can access the
     2108new HTML version from the notebook interface to \sage by clicking the
     2109\code{Help} link.
     2110
     2111Finally, you might want to share your changes with the \sage
     2112community.  To do this, use Mercurial (see Chapter~\ref{ch:mercurial})
     2113to produce patch files, and submit them to the \sage trac server.
     2114
     2115The \sage manuals and the corresponding files to edit:
     2116\begin{itemize}
     2117\item The \sage tutorial:
     2118\code{SAGE_ROOT/devel/doc/tut/tut.tex}
     2119\item The \sage programming guide:
     2120\code{SAGE_ROOT/devel/doc/prog/prog.tex}
     2121\item Constructions in \sage:
     2122\code{SAGE_ROOT/devel/doc/const/const.tex}
     2123\item The \sage installation guide:
     2124\code{SAGE_ROOT/devel/doc/inst/inst.tex}
     2125\item The \sage reference manual: some of this is contained in the file
     2126\code{SAGE_ROOT/devel/doc/ref/ref.tex}, but most of it is
     2127automatically generated from the \sage source code.  See the file
     2128\code{SAGE_ROOT/devel/doc/ref/README.txt} for information about how to
     2129edit the reference manual.
     2130\end{itemize}
     2131
     2132
     2133
     2134
     2135
     2136\part{Disseminating Code for \sage}
     2137
     2138\chapter{Introduction}
     2139
     2140Whether you've developed some new code for \sage or just have a simple
     2141bug fix, you need to know how to communicate what you've done to other
     2142\sage users.  This part of the guide discusses this issue.  Here are
     2143some of the available avenues of communication and tools to aid in
     2144that communication:
     2145
     2146\begin{itemize}
     2147\item the Google group \code{sage-support}, at
     2148\url{http://groups.google.com/group/sage-support}.  According to the
     2149group's official description, ``This group gives help and support to
     2150those who have problems with \sage (installation, syntax, etc).''
     2151\item the Google group \code{sage-devel}, at
     2152\url{http://groups.google.com/group/sage-devel}.  According to the
     2153group's description, ``This list is for \sage development and is about
     2154programming, design and technical issues.''  This is a great place to
     2155post questions about whether certain behavior is a bug, or whether a
     2156certain feature ought to be implemented (or how it ought to be
     2157implemented), or similar issues.
     2158\item Mercurial: this is the source control system that is included
     2159with \SAGE.  Use this to produce patches for \sage.  See
     2160Chapter~\ref{ch:mercurial} for a full discussion of Mercurial.
     2161\item the \sage trac server, at
     2162\url{http://trac.sagemath.org/sage_trac/}: this is where you should
     2163post bugs, patches for bugs, additions to the \sage library, etc.  See
     2164Chapter~\ref{ch:trac} for more information.
     2165\end{itemize}
     2166
     2167
     2168
     2169\section{Inclusion Procedure for New \sage Code}
     2170
     2171For code to become part of \sage's core, it must meet the
     2172following requirements:
     2173
     2174\begin{itemize}
     2175\item \textbf{License}. The license must be a GPL version 2+
     2176compatible license. (This will be publicly revisited around Jan 15,
     21772009.)
     2178\item \textbf{Build Support}.
     2179The code must build on our supported architectures and compilers
     2180(and intended port targets):
     2181\begin{itemize}
     2182\item Linux: x86, x86_64, Itanium, ppc, ppc64, Sparc (gcc 3.4-4.3)
     2183\item Apple Mac OS X: ppc, ppc64, x86, x86-64 (Xcode 2.5+)
     2184\item Microsoft Windows: x86, x86_64 MSVC 2005/Intel Fortran (MinGW or
     2185Cygwin support is insufficient!)
     2186\item Solaris 10: Sparc, x86, x86_64 (Sun Forte 12)
     2187\end{itemize}
     2188Remarks:
     2189\begin{itemize}
     2190\item Some \sage developers are willing to help you port to OSX,
     2191Solaris and Windows. But this is no guarantee and you or your project
     2192are expected to do the heavy lifting and also support those ports
     2193upstream if there is no \sage developer who is willing to share the
     2194burden.
     2195\end{itemize}
     2196Potential future ports include FreeBSD (x86, x86-64), OpenBSD (x86,
     2197x86-64), HPUX (Itanium), AIX (PPC64), and ARM (OSX).
     2198\item \textbf{Quality}.  The code should be ``better'' than any other
     2199available code (that passes the two above criteria), and the authors
     2200need to justify this. The comparison should be made to both Python and
     2201other software. Criteria in passing the quality test include:
     2202\begin{itemize}
     2203\item Speed
     2204\item Documentation
     2205\item Usability
     2206\item Memory leaks
     2207\item Maintainable
     2208\item Reasonable build time, size, dependencies
     2209\end{itemize}
     2210\item \textbf{Refereeing}.  The code must be refereed, as discussed in
     2211Chapter~\ref{ch:trac}.
     2212\end{itemize}
     2213
     2214
     2215
     2216
     2217\chapter{Producing Patches with Mercurial}
    32012218\label{ch:mercurial}
    32022219
    3203 Mercurial is the official source control system that is included with
    3204 \SAGE.  Mercurial is robust and works well even with huge data sets.
    3205 It's a solid piece of quality software.
    3206  
    3207     {\it All} the Mercurial repositories related to \SAGE are included
    3208     with \SAGE by default.  Thus the complete change history and setup
    3209     for doing development is available in your copy of \SAGE by
    3210     default.  It's not something that has to be confusingly installed
    3211     later.   \SAGE is designed to be extremely developer friendly.
     2220If you are editing or adding to \sage's core library, you will
     2221probably want to share your changes with other users.  Mercurial is
     2222the tool to do this.  Mercurial is the source control system that is
     2223included with \SAGE.  This chapter provides an overview of how to use
     2224Mercurial with \sage; see \url{http://www.selenic.com/mercurial/} for
     2225full documentation on Mercurial.
    32122226
    3213 \SAGE includes by default these Mercurial repositories:
     2227All of the Mercurial repositories related to \SAGE are included
     2228with \SAGE.  Thus the complete change history and setup
     2229for doing development is available in your copy of \SAGE.
     2230
     2231Before using Mercurial,
     2232make sure to define your username and password so the
     2233patches you make are identified as yours.  Make a
     2234file \verb+~/.hgrc+ in your home directory like this one:
     2235%skip
     2236\begin{verbatim}
     2237[ui]
     2238username = Cardinal Fang <fang@spanish-inquisition.com>
     2239\end{verbatim}
     2240
     2241
     2242
     2243\section{Quick Mercurial Tutorial for \SAGE}
     2244
     2245There are several ways to run Mercurial: from the command line, run
     2246\code{sage -hg} (\code{Hg} is the chemical symbol for mercury), or
     2247from within \sage, run \code{hg_sage}.  Most of the examples below use
     2248the second method.
     2249
     2250Before you modify \sage library files, you might want to create a copy
     2251of the \sage library in which to work.  Do this by typing \code{sage
     2252-clone myver}, for example; then \SAGE will use Mercurial to clone the
     2253current repository and call the result \code{myver}. The new
     2254repository is stored in \code{<SAGE_ROOT>/devel/sage-myver}, and when
     2255you clone, the symlink \verb+sage --> sage-myver+ is made.
     2256
     2257(You can also do, e.g., \code{sage -clone -r 1250 oldver}, to get a
     2258clone of \SAGE but as it was at revision 1250.  Of course, dependency
     2259issues could make old versions not work (e.g., maybe an old \SAGE
     2260library won't compile with the latest Singular library, which is what
     2261is installed elsewhere in SAGE_ROOT).  From with \sage, type
     2262\code{hg_sage.log()} to see the revision history.  \note{If you clone
     2263an old version, all of the Cython code is rebuilt, since there is no
     2264easy way to know which files do and don't need rebuilding.})
     2265
     2266Once you've copied the library to a new branch \code{myver} and edited
     2267some files there, you should build the \sage library to incorporate
     2268those changes: type \code{sage -b myver}, or just \code{sage -b} if
     2269the branch \code{myver} is already the current branch: that is, if
     2270\code{<SAGE_ROOT>/devel/sage} links to
     2271\code{<SAGE_ROOT>/devel/sage-myver}.  You can also type \code{sage -br
     2272myver} to build the library and then to immediately run \sage.
     2273
     2274If you want to submit your changes to the \sage development team for
     2275refereeing (and inclusion into \sage if the referee's report is
     2276positive), you should produce patch files.  To do this:
     2277
    32142278\begin{itemize}
    3215 \item {\tt SAGE_ROOT/devel/sage-*} -- the \SAGE library source code
    3216 \item {\tt SAGE_ROOT/devel/doc-*} -- the \SAGE documentation
    3217 \item {\tt SAGE_ROOT/data/extcode} -- new code included with \SAGE that is written for the systems \SAGE interfaces with, e.g., GAP, PARI, etc.
    3218 \item {\tt SAGE_ROOT/local/bin} -- the \SAGE shell scripts
     2279
     2280\item Type \code{hg_sage.status()} and \code{hg_sage.diff()} to see
     2281exactly what you've done (you can pass options to \code{diff} to see
     2282information about certain files).
     2283
     2284\item If you've added new files, not just edited existing ones, type
     2285\code{hg_sage.add([filenames])} to add those new files to your
     2286repository.
     2287
     2288\item Commit your changes by typing \code{hg_sage.commit([optional
     2289filenames])} to commit the changes in files to the repository --- if no
     2290filenames are given, all files are committed. First the output of
     2291\code{hg diff} is displayed: look at it or just enter \code{q}. Then
     2292you are dumped into an editor to type a brief comment on the changes.
     2293The default editor is vi, so type \code{i}, write some meaningful one
     2294line description, hit \code{Escape} and type \code{:wq}.  (In bash, to
     2295make emacs the default editor, type \code{export EDITOR=emacs}.)
     2296
     2297\item Now create a patch file using \code{hg_sage.export(...)}. This
     2298command needs a revision number (or list of revision numbers) as an
     2299argument; use \code{hg_sage.log()} to see these numbers. An optional
     2300second argument to \code{hg_sage.export(...)}  is a filename for the
     2301patch; the default is \code{(changeset_revision_number).patch}.
     2302
     2303\item Then post your patch on the \sage trac server: see
     2304Chapter~\ref{ch:trac}.
     2305
    32192306\end{itemize}
    32202307
    3221 Make sure to define your username and password so the
    3222 patches you make are identified as yours.  Make a
    3223 file \verb+~/.hgrc+ in your home directory like this one:
     2308Note that you can also start a very nice web server that allows you to
     2309navigate your repository with a web browser, or pull patches from it
     2310remotely, by typing \code{hg_sage.serve()}.
     2311
     2312Finally, if you want to apply a patch file (perhaps you've downloaded
     2313a patch from the trac server for review), use the command
     2314\code{hg_sage.patch('filename')}.
     2315
     2316
     2317
     2318\section{Using Mercurial with Other \sage Repositories}
     2319
     2320\SAGE includes these Mercurial repositories:
     2321\begin{itemize}
     2322\item {\tt SAGE_ROOT/devel/sage-*} : the \SAGE library source code
     2323\item {\tt SAGE_ROOT/devel/doc-*} : the \SAGE documentation
     2324\item {\tt SAGE_ROOT/data/extcode} : external system code, i.e., code
     2325included with \SAGE that is written for the systems with which \SAGE
     2326interfaces, e.g., GAP, PARI, etc.
     2327\item {\tt SAGE_ROOT/local/bin} : the \SAGE shell scripts
     2328\end{itemize}
     2329
     2330The previous section discussed using Mercurial with the \sage
     2331library, via the command \code{hg_sage}.  There are corresponding
     2332commands for each of the repositories:
     2333
     2334\begin{itemize}
     2335\item use \code{hg_sage} for the \sage library
     2336\item use \code{hg_doc} for the \sage documentation
     2337\item use \code{hg_extcode} for the external system code
     2338\item use \code{hg_scripts} for the \sage shell scripts
     2339\end{itemize}
     2340
     2341To produce patches for \sage's documentation, for example, use the
     2342same method as outlined above, but with commands like
     2343\code{hg_doc.status()}, \code{hg_doc.commit()}, etc.
     2344
     2345
     2346
     2347
     2348\chapter{Producing New \sage Packages}
     2349\label{ch:spkg}
     2350
     2351If you are producing code to add new functionality to \sage, you might
     2352consider turning it into a package (an \code{spkg} file) instead of a
     2353patch file.  If your code is very large (for instance) and should be
     2354offered as an optional download, a package is the right choice;
     2355similarly, if your code depends on some other optional component of
     2356\sage, you should produce a package.
     2357
     2358If you're not sure whether to build an spkg file or a patch file, ask
     2359for advice on \code{sage-devel}.
     2360
     2361
     2362
     2363
     2364\section{Creating a New spkg File}
     2365
     2366\sage packages are distributed as .spkg files, but an .spkg file is
     2367just a .tar.bz2 file (or a tar file), but named with .spkg to
     2368discourage confusion.  In particular, you can type
     2369%skip
    32242370\begin{verbatim}
    3225 [ui]
    3226 username = William Stein <wstein@gmail.com>
     2371     tar jxvf mypackage-version.spkg
     2372\end{verbatim}
     2373to extract one and see what is inside.
     2374
     2375Here's how to make your own spkg file:
     2376\begin{itemize}
     2377\item[(a)]
     2378Make a directory, e.g., \code{mypackage-0.1}.
     2379The name of the directory should be a lower-case string with no
     2380dashes, followed by a dash, followed by a version number.
     2381
     2382\item[(b)]
     2383Put your files in that directory.
     2384
     2385\item[(c)]
     2386Create an executable shell script \code{mypackage-0.1/spkg-install}.
     2387
     2388\item[(d)]
     2389The script \code{spkg-install} is run during installation
     2390of the \sage package.   In this script you may
     2391make the following assumptions:
     2392\begin{itemize}
     2393\item The PATH has the locations of \code{sage} and \code{python}
     2394(from the \sage installation) at the front.  Thus the command
     2395%skip
     2396\begin{verbatim}
     2397     python setup.py install
     2398\end{verbatim}
     2399will run the correct version of Python with everything set up
     2400correctly.  Also, running \code{gap} or \code{Singular}, for example,
     2401will run the correct version.
     2402
     2403\item
     2404The environment variable \code{SAGE_ROOT} points to
     2405the root directory of the \sage installation.
     2406\item
     2407The environment variable \code{SAGE_LOCAL} points to
     2408the \code{SAGE_ROOT/local} directory of the \sage installation.
     2409\item
     2410The environment variables
     2411\code{LD_LIBRARY_PATH} and \code{DYLD_LIBRARY_PATH}
     2412both have \code{SAGE_ROOT/local/lib} at the front.
     2413
     2414\end{itemize}
     2415
     2416\item[(e)]
     2417The \code{spkg-install} script should copy your files to the appropriate
     2418place after doing any build that is necessary.  That's it!
     2419
     2420\item[(f)] (Optional) Post a copy on the \sage trac server
     2421(Chapter~\ref{ch:trac}) so it can be refereed; if it gets a positive
     2422review, it might be included into the core \sage library, or it might
     2423become an optional download from the \sage web site, so anybody can
     2424automatically install it by typing \code{sage -i mypackage-version.spkg}.
     2425\end{itemize}
     2426
     2427If your package depends on another package, say GAP, then you should
     2428check that this other package has been installed.  In the case of GAP,
     2429it is installed into the directory \code{SAGE_ROOT/local/lib/gap-4.4.10/}
     2430and your \code{spkg-install} script should check that this exists,
     2431with the code like the following:
     2432%skip
     2433\begin{verbatim}
     2434if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.10 ]; then
     2435    echo "This package requires that GAP 4.4.10 is installed."
     2436    exit 1
     2437fi
    32272438\end{verbatim}
    32282439
    3229 \section{Quick Mercurial tutorial for \SAGE}
    3230  
    3231 Mercurial is available by default since \SAGE version 1.3.7.3.   It's confusingly
    3232 called {\tt hg} (the chemical symbol for mercury) at the command line.
    3233 Do "sage -hg" to run it, or make a link to \verb+SAGE_ROOT/local/bin/hg+
    3234  
    3235       The file \verb+<SAGE_ROOT>/local/lib/python/site-packages/sage+
    3236       is just a symbolic link to
     2440\emph{Caveat}: Do {\em not} just
     2441copy to \code{SAGE_ROOT/local/lib/gap*/} since that will copy
     2442your package to the lib directory of the {\em old} version of GAP if
     2443GAP is upgraded.
    32372444
    3238           \verb+<SAGE_ROOT>/devel/sage/build/sage+
     2445External Magma code goes in
     2446\verb+SAGE_ROOT/data/extcode/magma/user+, so if you want to redistribute
     2447Magma code with \sage as a package that Magma-enabled users can use,
     2448that's where you'd put it.  You'd also want to have relevant Python code
     2449to make the Magma code easily usable.
    32392450
    3240       Thus by changing what\verb+ <SAGE_ROOT>/devel/sage+ points to,
    3241       you can easily switch between running different versions of the
    3242       \SAGE library.  The "sage -b" command changes this symbolic link
    3243       for you automatically. Without argument "sage -b" switches to
    3244       where the symbolic link \verb+sage+ points to and builds \SAGE
    3245       in this branch.
     2451\textbf{Example}:
     2452These instructions provide directions for creating a particular
     2453\sage package.
    32462454
    3247       If you type \code{sage -clone myver}, say, then \SAGE will use
    3248       hg to clone the current repository and call the result myver.
    3249       The new repository is stored in \verb+<SAGE_ROOT>/devel/sage-myver+
    3250 
    3251       and when you clone the symlink \verb+sage --> sage-myver+
    3252 
    3253       is made.  Creating clones of a repository should be fairly fast,
    3254       e.g., about 30 seconds.
    3255       You can also do, e.g.,
    3256 
    3257                  \verb+sage -clone -r 1250 oldver+
    3258 
    3259       to get a clone of \SAGE but as it was at revision 1250.   Of course,
    3260       dependency issues could make old versions not work (e.g., maybe an
    3261       old \SAGE library won't compile with the latest Singular library, which
    3262       is what is installed elsewhere in SAGE_ROOT).
    3263       Type \code{hg_sage.log()} to see
    3264       the revision history.   
    3265       \note{All the cython code is rebuilt
    3266       if you clone an old version, since there is no easy way to
    3267       know which files do and don't need rebuilding.}
    3268  
    3269       To switch to a different repository and build it, type, e.g.,
    3270                    \verb+sage -b myver+
    3271       if the repository is stored in
    3272 
    3273                \verb+<SAGE_ROOT>/devel/sage-myver+
    3274 
    3275       To switch back type \verb+sage -b main+ to the repository stored in
    3276       \verb+<SAGE_ROOT>/devel/sage-main+
    3277  
    3278 
    3279 You can also use \code{hg} directly from the command line
    3280 instead of using it via the \code{hg_sage} object. 
    3281 
    3282 Use \code{sage -hg [options]} from within the repository.
    3283 See \code{hg(1)} in the man pages for option details.
    3284  
    3285 If you want to do a bunch of development, then make
    3286 it available to others, here is how:
    32872455\begin{itemize}
    32882456\item
    3289 Make a new branch (using \code{sage -clone myver}).
     2457Download \code{guava3.6.tar.gz} (the tarball for GUAVA, the
     2458GAP error-correcting codes package) from
     2459\url{http://www.gap-system.org/Packages/guava.html}.
     2460
    32902461\item
    3291 Implement your functionality there.
     2462Suppose you have a directory \code{sagefiles}.
     2463Within that, create a subdirectory \code{guava-3.6}.
     2464Extract the tarball into the directory \code{guava-3.6}.
     2465
    32922466\item
    3293 Type \code{hg_sage.status()} and
    3294 \code{hg_sage.diff()} to see exactly what
    3295 you've done (you can pass options to diff to
    3296 see information about certain files).
     2467Create a file \code{guava-3.6/spkg-install} with the following
     2468contents:
    32972469%skip
    32982470\begin{verbatim}
    3299 --------------------------------------------------------
    3300 | SAGE Version 1.6.1, Build Date: 2007-01-14           |
    3301 | Distributed under the GNU General Public License V2. |
    3302 --------------------------------------------------------
     2471#!/bin/sh
    33032472
    3304 Loading SAGE library. Current Mercurial branch is: myver
     2473if [ ! -d $SAGE_ROOT/local/lib/gap-4.4.10 ]; then
     2474    echo "This package requires that GAP 4.4.10 is installed."
     2475    exit 1
     2476fi
    33052477
    3306 sage: hg_sage.status()
    3307 Getting status of modified or unknown files:
    3308 cd "/home/jaap/sage/devel/sage" && hg status
    3309 M sage/databases/sloane_functions.py
     2478cp -pr guava-3.6 $SAGE_ROOT/local/lib/gap-4.4.10/pkg/
    33102479
    3311 ---
    3312 
    3313 Branch: myver
    3314 
    3315 sage: hg_sage.diff()
    3316 cd "/home/jaap/sage/devel/sage" && hg diff  | less
     2480cd $SAGE_ROOT/local/lib/gap-4.4.10/pkg/guava-3.6
     2481./configure ../..
     2482make
    33172483\end{verbatim}
    33182484
    3319 \item Type \code{hg_sage.add([filenames])} to add new
    3320 files to your repository.
    3321 \item Type \code{hg_sage.commit([optional filenames])}
    3322 to commit the changes in files to the repository--if no filenames are given all
    3323 files are committed. First the output of hg diff is piped through less, look at
    3324 it or just enter "q". Then you are asked to give your comment on the changes.
    3325 Type "i", write some meaningful one liner, hit 'Escape' and type ":wq".
    3326 (Note: vi is the default editor. In bash, to make emacs the default, type
    3327 \code{export EDITOR=emacs}.)
    3328 %skip
    3329 \begin{verbatim}
    3330 sage: hg_sage.commit()
    3331 cd "/home/jaap/sage/devel/sage" && hg diff  | less
    3332 cd "/home/jaap/sage/devel/sage" && hg commit
    3333 \end{verbatim}
     2485Make it executable with \code{chmod +x spkg-install}.
     2486
    33342487\item
    3335 Pull your changes back into your main repository: Note the argument to
    3336 hg_sage.pull() can be the full name starting with a '/' or the branch
    3337 name 'myver'.
     2488In the directory \code{sagefiles},
     2489issue the command
    33382490
    33392491%skip
    33402492\begin{verbatim}
    3341 sage -br main
     2493    sage -pkg guava-3.6
     2494\end{verbatim}
     2495This will do some checks, run tar/bz2, and create the spkg file,
     2496\code{guava-3.6.spkg}, in the
     2497directory \code{sagefiles},
     2498\end{itemize}
    33422499
    3343 sage: hg_sage.pull('myver') #or hg_sage.pull('/home/jaap/sage/devel/sage-myver')
    3344 cd "/home/jaap/sage/devel/sage" && hg status
    3345 cd "/home/jaap/sage/devel/sage" && hg status
    3346 cd "/home/jaap/sage/devel/sage" && hg pull -u /home/jaap/sage/devel/sage-myver
    3347 pulling from /home/jaap/sage/devel/sage-myver
    3348 searching for changes
    3349 adding changesets
    3350 adding manifests
    3351 adding file changes
    3352 added 1 changesets with 1 changes to 1 files
    3353 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
    3354 If it says use 'hg merge' above, then you should
    3355 type hg_sage.merge(), where hg_sage is the name
    3356 of the repository you are using.  This might not
    3357 work with the notebook yet.
     2500To test this, within the \code{sagefiles} directory, type
     2501%skip
     2502\begin{verbatim}
     2503    sage -i guava-3.6.spkg
     2504\end{verbatim}
     2505It should install and compile the guava program in sage.
     2506To test its functionality, one could do the following:
    33582507
    3359 sage: hg_sage.update()
    3360 cd "/home/jaap/sage/devel/sage" && hg update
    3361 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
     2508\begin{verbatim}
     2509sage: gap(`LoadPackage("guava")')          # requires optional package
     2510true
     2511sage: gap(`HammingCode(3,3)')              # requires optional package
     2512a linear [13,10,3]1 Hamming (3,3) code over GF(3)
    33622513\end{verbatim}
    33632514
    3364 \item
    3365 You can make {\it all} changes in the repository you're working in
    3366 as a bundle by typing \code{hg_sage.bundle('mybundle')} (this
    3367 creates an hg bundle \code{mybundle.hg}).  Alternatively,
    3368 you can export any particular changeset as plain text patches by
    3369 typing \code{hg_sage.export(...)}; note that each individual changeset
    3370 is recorded as a different patch.
    3371 \code{hg_sage.export(...)} needs at least the argument
    3372 revs -- integer or list of integers (revision numbers); use the hg_sage.log()
    3373 function to see them. An optional second argument is a 'patch_filename',
    3374 default is '(changeset_revision_number).patch'.
    3375 
    3376 These can be applied to any other
    3377 repository using \code{hg_sage.import_patch('patch_filename')}.
    3378  
    3379 Besides \code{hg_sage.diff(...)}, another way to see what
    3380 changes are contained in a patch bundle is to use the command
    3381 \code{hg_sage.inspect('patchname.hg')}. This displays the
    3382 ``diff file'' as plain text using \code{less}.
    3383 
    3384 Note that you don't get to ``cherry pick'' what you send in a bundle.
    3385 Everything you recorded is sent.  Hence you might want to use the
    3386 cloning, etc. above to send only from your own sage-main repository.
    3387 Note also that if you have a repository, e.g., sage-mynewstuff
    3388 and it's all ready to use, you can do
    3389                \verb+sage -hg pull <SAGE_ROOT>/devel/sage-mynewstuff+
    3390 from your \verb+<SAGE_ROOT>/devel/sage-main repository+, and you'll
    3391 get all the sage-mynewstuff merged in.   NOTE: read the output!
    3392 After that you must type \code{sage -hg update}.
    3393  
    3394 \item
    3395   You can also start a very nice web server that allows you to
    3396   navigate your repository with a web browser, or pull patches from it
    3397   remotely:
    3398 
     2515To  build it again if you've already installed the package, use
    33992516%skip
    34002517\begin{verbatim}
    3401 sage: hg_sage.serve()
     2518    sage -f guava-3.6.spkg
    34022519\end{verbatim}
    34032520
    3404 \item
    3405   For the occasional contributor:
    3406 %skip
    3407 \begin{verbatim}
    3408 sage: hg_sage.ci()
    3409 sage: hg_sage.send('patchname.hg')
    3410 \end{verbatim}
    34112521
    3412 Post the file 'patchname.hg' to the trac server.
    3413 Note that 'ci()' is an alias for 'commit()'.
     2522
     2523
     2524\chapter{The \sage Trac Server: Submitting Patches and Packages}
     2525\label{ch:trac}
     2526
     2527What should you do with your Mercurial patches for \sage?  You should
     2528post them on the \sage trac server.
     2529
     2530The \sage trac server, located at
     2531\url{http://trac.sagemath.org/sage_trac/}, is where \sage bugs are
     2532listed and patched, new code is posted and reviewed, and ideas for
     2533extending and improving \sage are discussed.  Thus if you find a bug
     2534in \sage, or if you have new code to submit, or if you have
     2535corrections for the documentation, you should post on the trac server.
     2536
     2537Items on the server are called ``tickets'', and anyone may browse the
     2538tickets: just visit \url{http://trac.sagemath.org/sage_trac/report}.
     2539You need to open an account, though, if you want to comment on a
     2540ticket, submit a patch, or create a new ticket.  See the web page
     2541\url{http://wiki.sagemath.org/TracGuidelines} for more information
     2542about obtaining an account, as well as general guidelines for using
     2543the trac server.
     2544
     2545
     2546
     2547\section{Reporting Bugs}
     2548
     2549``The first step is admitting you have a problem.''
     2550
     2551If you think you've found a bug in \sage, you should first search
     2552through the Google groups \code{sage-devel} and \code{sage-support}
     2553for postings related to your possible bug: maybe it has already been
     2554discussed.  You should also search the trac server to see if anyone
     2555else has opened a ticket about your bug.
     2556
     2557If you don't find anything, and you're not sure that you've found a
     2558bug, ask about it on \code{sage-devel}.
     2559
     2560If you don't find anything, and if you're positive you've found a bug,
     2561open a new ticket on the trac server.  As mentioned above, you need an
     2562account to do this.  To report a bug: log in and click on the ``New
     2563ticket'' button.  Type a meaningful one-liner in the ``Short summary''
     2564box, with more information in the larger box below.  You should
     2565include an explicit, reproducible example illustrating your bug
     2566(and/or the steps required to reproduce the buggy behavior).  You
     2567should also include the version of \sage (and any relevant packages)
     2568you are using, and operating system information, being precise as
     2569possible (32-bit, 64-bit, ...).
     2570
     2571Between the ``Summary'' and ``Full description'' boxes, there is a
     2572place to choose the ``Type'' of the ticket: ``Defect'',
     2573``Enhancement'', or ``Task''.  Use your best judgment here; a bug
     2574should probably be reported as a ``Defect''.
     2575
     2576Choose a priority for your bug, keeping in mind that the ``blocker''
     2577label should be used very sparingly.  Also pick a component for your
     2578bug; this is sometimes straightforward -- if your bug deals with
     2579\sage's calculus implementation, choose ``calculus''.  If it is not
     2580obvious, do your best.  Choose a milestone; if you're not sure what to
     2581choose, just choose the numbered version of sage from the menu
     2582(``sage-3.1.4'', for example).  Type in some helpful keywords.  In the
     2583box labeled ``Assign to'', type ``somebody'' if you're not sure what
     2584else to do.
     2585
     2586Hit the ``Preview'' button to make sure everything looks okay, and
     2587then hit ``Submit ticket''.
     2588
     2589
     2590
     2591\section{Guidelines for Opening Tickets}
     2592
     2593In addition to bug reports, you should also open a ticket if you have
     2594some new code which extends \sage's capabilities.  If you have a
     2595feature request, start a discussion on \code{sage-devel} first, and
     2596then if there seems to be general agreement that you have a good idea,
     2597open a ticket describing the idea.
     2598
     2599Other comments:
     2600\begin{itemize}
     2601\item Before opening a ticket, make sure that nobody else has opened a
     2602ticket about the same or closely related issue.
     2603\item It is much better to open several specific tickets than one that
     2604is very broad.  Indeed, a single ticket which deals with lots of
     2605different issues can be quite problematic, and should be avoided.
     2606\item Be precise: If foo doesn't work on OS X but is fine on Linux,
     2607mention that in the title; also use the keyword option so that
     2608searches will pick up the issue.
     2609\item The problem described in the ticket must be solvable.  For
     2610example, it would be silly to open a ticket whose purpose was ``Make
     2611\sage the best mathematical software in the world''. There is no metric
     2612to measure this properly and it is highly subjective.
    34142613\end{itemize}
    3415 You can browse the official repositories for \SAGE with your web browser here:
    3416     \verb+http://sagemath.org/sage/hg+
    3417 These are served using a cgi-bin script.
    3418  
    34192614
    3420 \section{Updating To the Latest Official \SAGE Library Source Code}
    34212615
    3422 If you're wary of messing things up, before typing the commands
    3423 explained below, first type from the command line
    3424 \code{sage -clone test}.  To switch back to your main repository (the default),
    3425 type \code{sage -b main}.
    34262616
    3427 To update to the latest \SAGE Mercurial source
    3428 code type \code{hg_sage.pull()}.  You may have to type
    3429 \code{hg_sage.update()}.
     2617\section{Patching Bugs/Working on Tickets}
    34302618
    3431 At this stage, you might get a message like:
    3432 \begin{verbatim}
    3433 remote changed .hgtags which local deleted
    3434 (k)eep or (d)elete?   
    3435 \end{verbatim}
    3436 Type \verb+k <RETURN>+. You might then get
    3437 \begin{verbatim}
    3438 remote changed export which local deleted
    3439 (k)eep or (d)elete?
    3440 \end{verbatim}
    3441 Type \verb+k <RETURN>+.
     2619If you have code which fixes a bug or deals with some issue in \sage,
     2620here's what to do: first, use Mercurial to create a patch file.  If
     2621the issue has been reported as a ticket on the trac server, attach your
     2622patch file to that ticket: go to the ticket, click on the ``Attach
     2623File'' button, and follow the directions.  On the ticket page, you
     2624should add a comment explaining your patch, and you should also
     2625change the summary for the ticket from \texttt{description of bug} to
     2626\texttt{[with patch, needs review] description of bug}.
    34422627
    3443 You might then get
    3444 \verb+merging sage/...+
    3445 Now kdiff3 opens with some text highlighted
    3446 in yellow and red. The bar at the bottom says "Number of remaining
    3447 unsolved conflicts: ...".
    3448 The conflict appears to be in the file named in the lower part
    3449 of the kdiff3 window. Clicked on "merge" on the
    3450 top bar, pick one of the options in the pop-down  menu,
    3451 then click on "save" and quit.
     2628If there is no trac ticket associated to this issue, create one (as
     2629explained in the previous sections) describing the issue and your
     2630solution, attach your patch, and give it a summary of the form
     2631\texttt{[with patch, needs review] description of bug goes here}.
    34522632
    3453 If you don't have kdiff3, install it with \code{apt get kdiff3} or
    3454 \code{yum install kdiff3}.
    34552633
    3456 If it says you need to, type
    3457 \code{hg_sage.commit()}
    3458 to finish updating your mercurial repository. 
    34592634
     2635\section{Reviewing Patches}
    34602636
    3461 \chapter{Miscellaneous}
    3462 \section{Circular Imports}
    3463 This needs to be cleaned up, but an irc log about this common problem
    3464 is better than nothing:
    3465 \begin{verbatim}
    3466 ncalexan1: from sage.schemes.plane_curves.curve import Curve_generic
    3467 [5:08pm] ncalexan1: gives an ImportError
    3468 was389: You can import anything from a file *unless* you set yourself up to
    3469 have a *circular* import loop.
    3470 [5:11pm] was389: The interpreter actually runs through the code of the file,
    3471 so it's not possible to circularly import usig
    3472 [5:11pm] ncalexan1: That's possible.
    3473 [5:11pm] was389: from blah import foo
    3474 [5:11pm] ncalexan1: Oh.
    3475 [5:11pm] was389: You might have to either (1) do import
    3476 sage.schemes.plane_curves.curve and explicitly access
    3477 [5:12pm] was389: Curve_generic, or (2) wait to import Curve_generic until
    3478 where it is used.
    3479 \end{verbatim}
     2637All code that goes into \sage is peer-reviewed, to ensure that the
     2638conventions discussed in this manual are followed, to make sure that
     2639there are sufficient examples and doctests in the documentation, and
     2640to try to make sure that the code does, mathematically, what it is
     2641supposed to.
    34802642
    3481 \section{Weird Issues}
    3482 \begin{enumerate}
    3483 \item
    3484 If you see
    3485 \begin{verbatim}
    3486 > > - "*** glibc detected *** free(): invalid pointer: 0x0846a684 ***"
    3487 \end{verbatim}
    3488 when leaving \sage, you might wonder why.
     2643If someone (other than you) has posted a patch for a ticket on the
     2644trac server, you can review it.  Look at the patch (by clicking on the
     2645file name in the list of attachments) to see if it makes sense.
     2646Download it (from the window displaying the patch, see the
     2647``Download'' option at the bottom of the page).  Apply it (using
     2648\code{hg_sage.patch('filename')}, for example) to your copy of \sage,
     2649and build \sage with the new code by typing \code{sage -b} (for
     2650patches to .py files) or \code{sage -ba} (for patches to .pyx files).
    34892651
    3490 Martin Albrecht finally figured out how to solve this strange bug. The
    3491 scenario: If you write a PYREX wrapper for some C code which needs to
    3492 be linked to the GMP library and you load the extension you wrote
    3493 during runtime into \sage you'll get such error messages on exit or
    3494 even more strange behavior.
    3495  
    3496 This vanishes completely if you load your extension module during the
    3497 startup of \SAGE like the other library wrappers/PYREX code (e.g. in
    3498 \code{sage/libs/all.py}). This is because when you load the module the
    3499 GMP library gets somehow reinitialized and all your prior GMP
    3500 variables are not valid anymore. So when they are freed/cleared on
    3501 exit you get this error stated above. Some objects relying on GMP are
    3502 created during startup of \SAGE so you'll see this behavior as well if
    3503 the first thing you do on the prompt is to load your library.
     2652Now ask yourself questions like these:
     2653\begin{itemize}
     2654\item Does the new source code make sense?
     2655\item When you run it in \sage, does it fix the problem reported on
     2656the ticket?
     2657\item Does it fail to introduce any new problems?
     2658\item Is it documented sufficiently, including both explanations and
     2659doctests?  (This is \textbf{very} important: all code in \sage must
     2660have doctests, so even if the patch is for code which didn't have a
     2661doctest before, the new version must include one.)
     2662\item In particular, is there a doctest illustrating that the bug has
     2663been fixed?  If a function used to give the wrong answer and this
     2664patch fixes that, then if possible, it should include a doctest
     2665illustrating its new success.
     2666\end{itemize}
    35042667
    3505 \end{enumerate}
     2668If the answers to these and other such reasonable questions are yes,
     2669then you might want to give the patch a positive review.  On the main
     2670ticket page, write a comment in the box and change the summary from
     2671\texttt{[with patch, needs review] description of bug} to
     2672\texttt{[with patch, positive review] description of bug}.  If you
     2673feel there are issues with the patch, explain them in the comment box,
     2674and change the summary to \texttt{[with patch, negative review]
     2675description of bug}, or \texttt{[with patch, needs work]
     2676description of bug}, or \texttt{[with patch, positive review pending fixes]
     2677description of bug}, or something similar.  Browse the tickets on the
     2678trac server to see how things are done.
    35062679
    3507 \section{Benchmarking}
    3508 \label{sec:Benchmarking}
     2680By the way, if you review a patch which deals with the \sage manuals,
     2681say, instead of the source code, then you need to use
     2682\code{hg_doc.patch('filename')} instead of
     2683\code{hg_sage.patch('filename')} to apply it, and you need to follow
     2684the directions in Chapter~\ref{ch:manuals} to build the new
     2685documentation.
    35092686
    3510 \begin{verbatim}
    35112687
    3512 http://sage.math.washington.edu:8100/william_32bit_add?edit
     2688\section{Closing Tickets}
    35132689
    3514 {{{
    3515 if is_64_bit:
    3516     word = 64
    3517 else:
    3518     word = 32
    3519 }}}
     2690Don't close tickets.  That is the job of the \sage administrators.  If
     2691you feel strongly that a ticket should be closed or deleted, send
     2692email to the current release manager explaining the situation.
    35202693
    3521 {{{
    3522 def testpy(n, op):
    3523     a = long(ZZ.random(2^(word*n)))
    3524     b = long(ZZ.random(2^(word*n)))
    3525     N = 1000000
    3526     t = cputime()
    3527     for i in range(N):
    3528         c = op(a, b) 
    3529     return cputime(t)
    3530 }}}
    35312694
    3532 {{{
    3533 def testgmp(n, op):
    3534     a = ZZ.random(2^(word*n))
    3535     b = ZZ.random(2^(word*n))
    3536     N = 1000000
    3537     t = cputime()
    3538     for i in range(N):
    3539         c = op(a, b)
    3540     return cputime(t)
    3541 }}}
    35422695
    3543 {{{
    3544 op=operator.add
    3545 for n in range(1,20):
    3546     print 4*n, testpy(4*n,op) / testgmp(4*n,op)
    3547 ///
    3548 4 0.336633663366
    3549 8 0.378640776699
    3550 12 0.388349514563
    3551 16 0.450980392157
    3552 20 0.509803921569
    3553 24 0.509259259259
    3554 28 0.527272727273
    3555 32 0.504587155963
    3556 36 0.653846153846
    3557 40 0.590476190476
    3558 44 0.681818181818
    3559 48 0.718181818182
    3560 52 0.654867256637
    3561 56 0.648648648649
    3562 60 0.973451327434
    3563 64 0.956896551724
    3564 68 0.920353982301
    3565 72 1.07894736842
    3566 76 1.04347826087
    3567 }}}
    35682696
    3569 {{{
    3570 op=operator.mul
    3571 for n in range(1,10):
    3572     print n, testpy(n,op) / testgmp(n,op)
    3573 ///
    3574 1 0.475
    3575 2 0.5
    3576 3 0.527777777778
    3577 4 0.663716814159
    3578 5 0.801724137931
    3579 6 0.991596638655
    3580 7 1.21551724138
    3581 8 1.41176470588
    3582 9 1.56589147287
    3583 }}}
    35842697
    3585 \end{verbatim}
     2698\appendix
    35862699
    3587 \begin{verbatim}
    3588 
    3589 http://sage.math.washington.edu:8100/william_64bit_add
    3590 
    3591 
    3592 {{{
    3593 if is_64_bit:
    3594     word = 64
    3595 else:
    3596     word = 32
    3597 print word
    3598 ///
    3599 64
    3600 }}}
    3601 
    3602 {{{
    3603 def testpy(n, op):
    3604     a = long(ZZ.random(2^(word*n)))
    3605     b = long(ZZ.random(2^(word*n)))
    3606     N = 1000000
    3607     t = cputime()
    3608     for i in range(N):
    3609         c = op(a, b) 
    3610     return cputime(t)
    3611 }}}
    3612 
    3613 {{{
    3614 def testgmp(n, op):
    3615     a = ZZ.random(2^(word*n))
    3616     b = ZZ.random(2^(word*n))
    3617     N = 1000000
    3618     t = cputime()
    3619     for i in range(N):
    3620         c = op(a, b)
    3621     return cputime(t)
    3622 }}}
    3623 
    3624 {{{
    3625 op=operator.add
    3626 for n in range(1,20):
    3627     print 4*n, testpy(4*n,op) / testgmp(4*n,op)
    3628 ///
    3629 4 0.478991596639
    3630 8 0.5
    3631 12 0.526717557252
    3632 16 0.55905511811
    3633 20 0.5859375
    3634 24 0.615384615385
    3635 28 0.709923664122
    3636 32 0.731884057971
    3637 36 0.833333333333
    3638 40 0.873134328358
    3639 44 0.878571428571
    3640 48 0.85401459854
    3641 52 0.950704225352
    3642 56 1.1
    3643 60 1.12121212121
    3644 64 0.977941176471
    3645 68
    3646 }}}
    3647 
    3648 {{{
    3649 op=operator.mul
    3650 for n in range(1,10):
    3651     print n, testpy(n,op) / testgmp(n,op)
    3652 ///
    3653 1 0.487179487179
    3654 2 0.747826086957
    3655 3 1.13913043478
    3656 4 1.56896551724
    3657 5 2.0859375
    3658 6 2.63846153846
    3659 7 3.35384615385
    3660 8 4.38636363636
    3661 9 5.0962962963
    3662 }}}
    3663 
    3664 
    3665 \end{verbatim}
    3666 
    3667 \chapter{History and License}
     2700\chapter{\sage's License}
    36682701
    36692702\input{license}
    36702703