# HG changeset patch
# User J. H. Palmieri
# Date 1219199147 25200
# Node ID 8f9275ea4358f44711241bbc022128ef6a46d202
# Parent bba79837cb391923027900edb059116505472ab7
revising programming guide
diff r bba79837cb39 r 8f9275ea4358 commontex/macrosnew.tex
 a/commontex/macrosnew.tex Tue Aug 19 19:25:13 2008 0700
+++ b/commontex/macrosnew.tex Tue Aug 19 19:25:47 2008 0700
@@ 45,6 +45,12 @@
% when run through latex2html. See the command \newbf above for an
% application of this. See the latex2html docs for more information.
\usepackage{html}
+
+\usepackage{verbatim}
+% the following is used for nesting verbatim environments, used in prog/prog.tex
+\newenvironment{newverbatim}%
+ \verbatim%
+ {\endverbatim}
\newcommand{\by}[1]{\par\noindent{\bf AUTHOR:} #1\par}
diff r bba79837cb39 r 8f9275ea4358 prog/prog.tex
 a/prog/prog.tex Tue Aug 19 19:25:13 2008 0700
+++ b/prog/prog.tex Tue Aug 19 19:25:47 2008 0700
@@ 1,18 +1,24 @@
+% see 3393, 3882
+
+% coercion section
+% mutability section should be rewritten, expanded.
+% benchmarking?
+
\documentclass{manual}
\usepackage{textcomp}
\include{macros}
+
+\include{macrosnew}
% a hack to get around another hack.
\ifpdf
\usepackage{url}
\else
\fi

%\newcommand{\todo}[1]{\footnote{1}}
\newcommand{\todo}[1]{}
\newcommand{\swlist}{GAP, GSL, Matplotlib, Maxima, MWRANK, NetworkX,
 NTL, Numpy, PARI and Singular}
+ NTL, Numpy, PARI and Singular\xspace}
\usepackage{color}
\definecolor{hue}{rgb}{.202, .602, .58}
@@ 31,11 +37,9 @@
\begin{document}
\title{\SAGE Programming Guide}
+\title{\SAGE Developer's Guide}
\author{William Stein%\thanks{Associate Professor, University of Washington, wstein@math.washington.edu.}
\ \ and David Joyner%\thanks{Mathematics Department, US Naval Academy, wdj@usna.edu}
}
+\author{William Stein and David Joyner}
\maketitle
@@ 48,26 +52,28 @@
\begin{abstract}
\noindent
Absolutely {\bf everybody} who uses \SAGE should contribute something
+\SAGE is a free mathematics software system. It is implemented using
+Python, Cython, and C++, and uses \swlist. It is free and open
+source, and is available under the terms of the GNU Public License.
+
+Everybody who uses \SAGE should contribute something
back to \SAGE at some point. Implement a new function, add examples
to the documentation, find bugs and typos, fix a bug, create a new
class, create a fast new C library, etc. This book is a guide about
how to contribute code back to \SAGE.
+how to contribute to \SAGE.
This document describes how to write programs using \SAGE, and also
how to modify and extend the core \SAGE libraries.

\SAGE stands for Software for Algebra and Geometry Experimentation.
It is implemented using Python, Cython, and C++, and uses \swlist.
It is free and open source, and is available under the terms of the
GNU Public License.
+This document describes how to write programs using \SAGE,
+how to modify and extend the core \SAGE libraries, and how to modify
+\sage's documentation. It also discusses how to share your new and
+modified code with other \SAGE users.
\end{abstract}
\tableofcontents
\todo{Remove I and me}
+
+\part{Writing Code for \sage}
\chapter{Introduction}
If there is something you would like to implement and make
@@ 93,13 +99,12 @@
contains sophisticated, optimized number theory algorithms. Notably
absent from this triad is a good system for exact linear algebra
(something MAGMA does extremely well), but this gap is being filled by
code being written for \SAGE. \todo{Say something about GSL,
 Matplotlib, Maxima, MWRANK, NetworkX, NTL, Numpy}
%Unfortunately, Singular, GAP, and PARI
%have vastly different languages and conventions.
+code written for \SAGE.
+\todo{Say something about GSL, Matplotlib, Maxima, MWRANK, NetworkX,
+NTL, Numpy}
\SAGE is not just about gathering together functionality (something
UNIX already does well), it is about providing a clear, systematic and
+\SAGE is not just about gathering together functionality, it is about
+providing a clear, systematic and
consistent way to access a large number of algorithms, in a coherent
framework that makes sense mathematically. In the design of \SAGE, the
semantics of objects, the definitions, etc., are informed by how the
@@ 108,14 +113,20 @@
This document was authored by William Stein, David Joyner and others
with the editorial help of Iftikhar Burhanuddin.
\chapter{\SAGE{} Coding Conventions}\label{ch:conventions}
To meet the goal of making \sage easy to read, all Python/Cython code
that is included with \sage should adhere to the style conventions
mentioned in this chapter.
\section{Coding Conventions}
Follow the standard Python formatting rules when writing code for Sage,
as explained at \url{http://www.python.org/doc/essays/styleguide.html}.
+
+\chapter{Conventions for Coding in \SAGE}\label{ch:conventions}
+
+To meet the goal of making \sage easy to read, maintain, and improve,
+all Python/Cython code that is included with \sage should adhere to
+the style conventions discussed in this chapter.
+
+\section{Python Coding Conventions}
+
+Follow the standard Python formatting rules when writing code for \sage,
+as explained at
+\url{http://www.python.org/dev/peps/pep0008/} and
+\url{http://www.python.org/dev/peps/pep0257/}.
In particular,
\begin{itemize}
\item Use 4 spaces for indentation levels. Do not use tabs as they
@@ 128,7 +139,7 @@
\begin{verbatim}
def set_some_value()
\end{verbatim}
 instead of:
+ instead of CamelCase:
\begin{verbatim}
def SetSomeValue()
\end{verbatim}
@@ 137,31 +148,26 @@
\end{itemize}
\section{File and Directory names}
+\section{File and Directory Names}
+
Python \sage library code uses the following conventions. Directory
names may be plural (e.g., \code{rings}) and file names are almost
always singular (e.g., \code{polynomial\_ring.py}). Note that the
file \code{polynomial\_ring.py} might still contain definitions of
several different types of polynomial rings.
\note{You are strongly encouraged
+\note{You are encouraged
to include miscellaneous notes, emails, design
discussions, etc., in your package. Make these
plain text files (with extension \code{.txt})
in a subdirectory directory called \code{notes}.
+in a subdirectory called \code{notes}. (For example, see
+\code{SAGE_ROOT/devel/sage/sage/ext/notes/}.)
}
%\section{Ordering of function names in a class}
%Function names
%should be defined in alphabetical order, with \code{__}'s first,
%then \code{_}'s, then public functions. This is for easy of readability
%and finding code. It also encourages functions with related behavior
%to be named in a similar way, which makes it easier to find them
%in the documentation and when using tab completion from the interpreter.

\section{Headings of \SAGE Library Code Files}
The top of each \sage code file should appear as follows\todo{What is \section{Tutorial} below}:
+
+The top of each \sage code file should follows this format:
\begin{verbatim}
r"""
@@ 171,34 +177,31 @@
...
AUTHORS:
  YOUR NAME (20050103): initial version
  person (date in ISO yearmonthday format): short desc
+  YOUR NAME (20050103): initial version
+  person (date in ISO yearmonthday format): short desc
...
  person (date in ISO yearmonthday format): short desc
+  person (date in ISO yearmonthday format): short desc
\section{Tutorial}
...
Lots and lots of examples.
"""
#*****************************************************************************
# Copyright (C) 2006 William Stein
# 2006 YOUR NAME
+# Copyright (C) 2008 YOUR NAME
#
# Distributed under the terms of the GNU General Public License (GPL)
# http://www.gnu.org/licenses/
#*****************************************************************************
\end{verbatim}
The following is top of the ssmod.py file, which contains the
SupersingularModule code and is located at
\code{SAGE_ROOT/devel/sage/sage/modular/ssmmod/ssmod.py}.
+The following is the top of the file
+\code{SAGE_ROOT/devel/sage/sage/modular/ssmmod/ssmod.py}, which
+contains the Supersingular Module code.
{\small
\begin{verbatim}

"""
Module of Supersingular Points
@@ 216,25 +219,35 @@
[
(Vector space of degree 33 and dimension 1 over Finite Field of size 97
Basis matrix:
 [ 0 0 0 1 96 96 1 0 95 1 1 1 1 95 2 96 0 0 96 96 0 0 96
 2 96 96 2 0 0 1 1 95 0], 1),
+ [ 0 0 0 1 96 96 1 96 96 0 2 96 96 0 1 0 1 2 95 0 1 1 0 1 0 95 0 96 95 1 96 0 2], True),
(Vector space of degree 33 and dimension 1 over Finite Field of size 97
Basis matrix:
 [ 0 1 96 16 75 22 81 0 0 17 17 80 80 0 0 16 1 40 74 23 57 96 81
 0 23 74 0 0 0 24 73 0 0], 1),
+ [ 0 1 96 75 16 81 22 17 17 0 0 80 80 1 16 40 74 0 0 96 81 23 57 74 0 0 0 24 0 23 73 0 0], True),
(Vector space of degree 33 and dimension 1 over Finite Field of size 97
Basis matrix:
 [ 0 1 96 90 90 7 7 0 0 6 91 6 91 0 0 7 13 0 91 6 0 84 90
 0 91 6 0 0 0 90 7 0 0], 1)
+ [ 0 1 96 90 90 7 7 6 91 0 0 91 6 13 7 0 91 0 0 84 90 6 0 6 0 0 0 90 0 91 7 0 0], True)
]
sage: len(D)
9
We compute a Hecker operator on a space of huge dimension!
sage: X = SupersingularModule(next_prime(100000))
 sage: t = X.T(2).matrix() # long time (but still less than a minute!)
 sage: t.nrows() # long time
+ sage: t = X.T(2).matrix() # long time (but still less than a minute!)
+ sage: t.nrows() # long time
8334
+
+TESTS:
+ sage: X = SupersingularModule(389)
+ sage: T = X.T(2).matrix().change_ring(QQ)
+ sage: d = T.decomposition()
+ sage: len(d)
+ 6
+ sage: [a[0].dimension() for a in d]
+ [1, 1, 2, 3, 6, 20]
+ sage: loads(dumps(X)) == X
+ True
+ sage: loads(dumps(d)) == d
+ True
"""
#*****************************************************************************
@@ 248,15 +261,16 @@
}%small
All code included with \sage must be licensed under the GPL or a less
restrictive license (e.g., the BSD license). You should share
copyright with William Stein by including your name at the top. It is {\em very
+restrictive license (e.g., the BSD license).
+It is {\em very
important} that you include your name in the AUTHOR log, since
everybody who submits code to \sage to receive proper credit. (If
ever you feel you are not receiving proper credit for anything you
submit to \sage, please let William Stein know!)
+submit to \sage, please let the development team know!)
\section{Documentation Strings}\label{sec:convdocs}
{\bf Every} function should have a docstring that includes
+
+\textbf{Every} function must have a docstring that includes
the following information:
\begin{itemize}
\item A onesentence description of the function, followed by a blank line.
@@ 268,14 +282,16 @@
\item An EXAMPLES block for examples. This is {\em not} optional.
\item A NOTES block for special notes (optional). Include information
such as algorithm used, references, etc.
\item An AUTHORS block (important!).
+\item An AUTHORS block (optional, but encouraged for important
+functions, so users can see from the docstring who wrote it and
+therefore whom to contact if they have questions).
\end{itemize}
Use the following template when documenting functions. Note the
indentation:
%skip
\begin{verbatim}
def point(self, x=1, y=2):
 r""" # note the r for "raw string"
+ r"""
This function returns the point $(x^5,y)$.
INPUT:
@@ 321,7 +337,28 @@
\begin{itemize}
\item Use nice \Latex formating everywhere but in the
 \code{INPUT/OUTPUT} blocks.
+ \code{INPUT/OUTPUT} blocks. If you use backslashes, either use
+ double backslashes or place an {\tt r} right before the first triple
+ opening quote. For example,
+%skip
+\begin{verbatim}
+def cos(x):
+ """
+ Returns $\\cos(x)$.
+ """
+
+def sin(x):
+ r"""
+ Returns $\sin(x)$.
+ """
+\end{verbatim}
+
+Also, the nonstandard \Latex macros that are used when typesetting the
+documentation are defined in
+\code{SAGE_ROOT/doc/commontex/macros.tex}. If you need to add more
+macros, add them there and post a patch file on the \sage trac server
+(Chapter~\ref{ch:trac}) so that they can be included in the next
+version of \sage.
\item Liberally describe what the examples do. Note that there must
be a blank line after the example code and before the explanatory
@@ 334,23 +371,36 @@
regular basis, and are crucial for the quality and adaptability of
\sage. Without such examples, small changes to one part of \sage
that break something else might not go seen until much later when
 someone uses the system, which is frustrating.
+ someone uses the system, which is unacceptable.
\end{itemize}
The code in the examples should pass automatic testing. This means
that if the above code is in the file \code{f.py} (or \code{f.sage}),
then \code{sage t f.py} should not give any error messages. Testing
occurs with full \sage preparsing of input within the standard \sage
shell environment. {\em Important.}\todo{Clarify with an
 example/pathology and explanation. } The file \code{f.py} is not
+shell environment, as described in Section~\ref{sec:Preparsing}.
+{\em Important:} The file \code{f.py} is not
imported when running tests unless you have arranged that it be
imported into your \sage environment, i.e., unless its functions are
available when you start \sage using the \code{sage} command.
+available when you start \sage using the \code{sage} command. For
+example, the function \code{cdd_convert} in the file
+\code{SAGE_ROOT/devel/sage/sage/geometry/polyhedra.py} includes
+an EXAMPLES block containing the following:
+\begin{verbatim}
+ sage: from sage.geometry.polyhedra import cdd_convert
+ sage: cdd_convert(' 1 1 0 0')
+ [1, 1, 0, 0]
+\end{verbatim}
+\sage doesn't know about the function \code{cdd_convert} by default,
+so it needs to be imported before it is tested; hence the first line
+in the example.
\subsection{Further conventions for automated testing of examples}
+
+
+\subsection{Further Conventions for Automated Testing of Examples}
\label{sec:Furtherconventions}
The Python script \code{local/bin/sagedoctest} implements
documentation testing in \sage (see Chapter~\ref{ch:testing} for more
+documentation testing in \sage (see Section~\ref{ch:testing} for more
details). When writing documentation, keep the following points in
mind:
\begin{itemize}
@@ 359,9 +409,20 @@
\code{2/3} as a rational instead of the Python int \code{0}. For
more information on preparsing see Section \ref{sec:Preparsing}.
\item If a test line contains the text \code{random} it is executed by
+\item If a test line contains the text \code{random}, it is executed by
\code{sagedoctest} but \code{sagedoctest} does not check that the
output agrees with the output in the documentation string.
+ For example, the docstring for the \code{__hash__} method for
+ \code{CombinatorialObject} in
+ \code{SAGE_ROOT/devel/sage/sage/combinat/combinat.py} includes the
+ lines
+%skip
+\begin{verbatim}
+ sage: hash(c) #random
+ 1335416675971793195
+ sage: c._hash #random
+ 1335416675971793195
+\end{verbatim}
\item If a line contains the text \code{long time} then that line is
not tested unless the {\code long} option is given, e.g.,
@@ 370,30 +431,15 @@
\sage development, but will get run before major releases. No
example should take more than about 30 seconds.
 Here is the \code{Lseries} function docstring which contains the phrase
 \code{long time} and is taken from
\code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_rational.py}

+ For example, here is part of the docstring from the \code{regulator} method for
+ rational elliptic curves, from the file
+ \code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_rational.py}:
+%skip
\begin{verbatim}
 def Lseries(self, s):
 r"""
 Returns the value of the Lseries of the elliptic curve E at s, where s
 must be a real number.

 Use self.Lseries_extended for s complex.

 \note{If the conductor of the curve is large, say $>10^{12}$,
 then this function will take a very long time, since it uses
 an $O(\sqrt{N})$ algorithm.}

 EXAMPLES:
 sage: E = EllipticCurve([1,2,3,4,5])
 sage: E.Lseries(1)
 0.000000000000000
 sage: E.Lseries('1.1') # long time (!)
 0.28549100767814833
+ sage: E = EllipticCurve([0, 0, 1, 1, 0])
+ sage: E.regulator() # long time (1 second)
+ 0.0511114082399688
\end{verbatim}

\item If a line contains \code{todo: not implemented}, it is never
tested. It is good to include lines like this to make clear
@@ 409,10 +455,13 @@
\code{sage t optional f.py}. (Note that \code{optional}
must not be the first argument to \code{sage}.) Use this
to include doctests that require optional packages.
For example,
+For example, the docstring for \code{_magma_init_} in the
+class \code{EllipticCurve_finite_field} from the
+\code{SAGE_ROOT/devel/sage/sage/schemes/elliptic_curves/ell_finite_field.py}
+contains
\begin{verbatim}
 sage: E.padic_regulator(5) # requires optional MAGMA package
 xxx
+ sage: EllipticCurve(GF(41),[2,5])._magma_init_() # optional  requires Magma
+ 'EllipticCurve([_sage_[1]GF(41)!0,GF(41)!0,GF(41)!0,GF(41)!2,GF(41)!5])'
\end{verbatim}
\item If the entire documentation string contains all three words
@@ 424,40 +473,139 @@
\end{itemize}
Using \code{sage_search} from the \sage prompt or \code{grep} one can
easily find aforementioned keywords and in the case of \code{todo: not
 implemented} use the results to motivate further development on \sage.
+Using \code{search_src} from the \sage prompt (or \code{grep}), one can
+easily find the aforementioned keywords. In the case of \code{todo: not
+ implemented}, one can use the results of such a search to direct
+further development on \sage.
\section{Exceptions}
Whenever code such as the following should be avoided:
+
+\section{Automated Testing}\label{ch:testing}
+
+This section describes \sage's automated testing of test files of the
+following types: \code{.py .pyx .sage .tex}. Briefly, use \code{sage t
+ } to test that the examples in \code{} behave exactly as
+claimed. See the following subsections for more details.
+See also Section~\ref{sec:convdocs} for a discussion of how to
+include examples in documentation strings and what conventions to
+follow.
+
+\subsection{Testing .py, .pyx and .sage Files}\label{sec:test:python}
+
+Run \code{sage t } to test that all code examples in
+\code{filename.py}. Similar remarks apply to \code{.sage} and
+\code{.pyx} files.
+
+\begin{verbatim}
+ sage t [verbose] [optional] [files and directories ... ]
+\end{verbatim}
+
+When you run \code{sage t }, \sage makes a copy of
+\code{} with all the \code{sage} prompts replaced by
+\code{{>}{>}>}, then uses the standard Python doctest framework to test
+the documentation. More precisely,
+the Python script \code{local/bin/sagedoctest} implements
+documentation testing. It does the following when asked
+to test a file \code{foo.py} or \code{foo.sage}.
+\begin{enumerate}
+
+\item Creates the directory \code{.doctest} if it doesn't
+exist and the file \code{.doctest/foo.py}.
+
+\item The file \code{.doctest_foo.py} contains functions for
+each docstring in \code{foo.py}, but with all \sage preparsing
+applied and with \code{from sage.all import *} at the top.
+The function documentation is thus standard Python with
+\code{{>}{>}>} prompts.
+
+\item \code{sagedoctest} then runs \sage's Python interpreter
+on \code{.doctest_foo.py}.
+\end{enumerate}
+
+\note{{\em Note that line numbers in the errors
+you might see apply to the file \code{.doctest_foo.py}, not
+to the original file \code{foo.py}!} If you get errors,
+feel free to inspect \code{.doctest_foo.py}; it is never
+automatically deleted by \sage.}
+
+Your file passes these tests if the code in it will run when entered
+at the \code{sage:} prompt with no special imports. Thus users are
+guaranteed to be able to exactly copy code out of the examples you
+write for the documentation and have them work.
+
+
+
+\subsection{Testing \Latex Documentation}
+
+Run \code{sage t } to test the examples in
+verbatim environments in \Latex documentation. Put
+\code{\%skip}
+on the line right before a verbatim environment to have that
+example skipped when testing the file.
+
+\sage creates a file \code{.doctest_filename.py} and tests
+it just as for \code{.py}, \code{.pyx} and \code{.sage}
+files. In order to skip testing of a block of code
+in a verbatim environment, put the \Latex comment
+\code{\%skip} on the previous line.
+
+Of course in \Latex files, one
+often inserts explanatory texts between different
+verbatim environments. To link together verbatim
+environments use the \code{\%link} comment. For
+example:
%skip
\begin{verbatim}
try:
 some code
except: # bad
 more code
\end{verbatim}
Instead catch specific exceptions. For example,
\begin{verbatim}
try:
 return self.__coordinate_ring
except (AttributeError, other exceptions), msg: # Good
 more code to compute something
\end{verbatim}
Note that the syntax in the except is to list all the exceptions
that are caught as a tuple, followed by the variable that holds
the message.
+\begin{newverbatim}
+ \begin{verbatim}
+ sage: a = 1
+ \end{verbatim}%link
If you don't have any exceptions explicitly listed (as a tuple), your
code will catch absolutely anything, including ctrlC and alarms,
and this will lead to confusion.
+ Next we add 1 to \code{a}.
+ %link
+ \begin{verbatim}
+ sage: 1 + a
+ 2
+ \end{verbatim}
+\end{newverbatim}
\section{Optional Packages} If an optional package is required for a
certain function to work, that function should fail gracefully 
perhaps using a \code{try} and \code{except} block  when the optional
package is not available, and should give a hint about how to install
it. For example, typing \code{sage optional} gives a list of all
optional packages, so suggest to the user that they type that.
+See \code{SAGE_ROOT/doc/tut/tut.tex} for many examples
+of how to include automated testing in \Latex documentation
+for \sage.
+
+
+
+\section{Randomized Testing}
+\label{ch:randomtesting}
+
+In addition to all the examples in your docstrings, which serve as
+both demonstrations and tests of your code, you should consider creating a
+test suite. Think of this as a program that will run for a while and
+``tries'' to crash your code using randomly generated input. Your
+test code should define a class \class{Test} with a {\tt random()}
+method that runs random tests. These are all assembled together
+later, and each test is run for a certain amount of time on a regular
+basis.
+
+For example, see the file \code{SAGE_ROOT/devel/sage/sage/modular/modsym/tests.py}.
+
+
+
+\chapter{Coding in Python for \SAGE}
+
+This chapter discusses some issues with, and advice for, coding in \SAGE.
+
+
+\section{Design}
+
+If you are planning to develop some new code for \sage, design is
+important. So think about what your program will do and how that fits
+into the structure of \SAGE. In particular, much of \SAGE is
+implemented in the objectoriented language Python, and there is a
+hierarchy of classes that organize code and functionality. For
+example, if you implement elements of a ring your class should derive
+from \class{sage.structure.element.RingElement}, rather than starting
+from scratch. Try to figure out how your code should fit in with
+other \sage code, and design it accordingly.
+
\section{Special \sage Functions}
@@ 472,7 +620,7 @@
also has special methods. They are typically of the form
\code{_XXX_}. (In a few cases the trailing underscore is not
included, but this will be changed so that the trailing underscore is
always included.) This section describes all special methods.
+always included.) This section describes these special methods.
All objects in \sage should derive from the Cython extension
class \code{SageObject}:
@@ 483,6 +631,14 @@
class MyClass(SageObject,...):
...
\end{verbatim}
+or from some other already existing \sage class:
+%skip
+\begin{verbatim}
+from sage.rings.ring import Algebra
+
+class MyFavoriteAlgebra(Algebra):
+ ...
+\end{verbatim}
You should implement the \code{\_latex\_} and \code{\_repr\_} method
for every object. The other methods depend on the nature of the
@@ 491,19 +647,18 @@
\subsection{\Latex Representation}
Every object x in \sage should support the command \code{latex(x)}, so
that any \sage object can be easily and accurately displayed via
\Latex. Here is how to make a class and therefore its instances
+\Latex. Here is how to make a class (and therefore its instances)
support the command \code{latex}.
\begin{enumerate}
\item Put \code{import sage.misc.latex as latex} at the top of the
 file that defines your class.

\item Define a member function \code{_latex_(self)} that returns a
+\item Define a method \code{_latex_(self)} that returns a
\Latex representation of your object. It should be something that
can be typeset correctly within math mode.
\item Often objects are built up out of other options. For example if
+\item Often objects are built up out of other \sage objects, and these
+ components should be typeset using the \code{latex} function.
+ For example if
\code{c} is a coefficient of your object, and you want to typeset
\code{c} using latex, use \code{latex(c)} instead of
\code{c._latex_()}, since \code{c} might not have a \code{_latex_}
@@ 513,19 +668,15 @@
illustrates latex generation for your object.
\item You can use any macros included in \code{amsmath},
 \code{amssymb}, \code{amsfonts} or the ones defined in
+ \code{amssymb}, or \code{amsfonts}, or the ones defined in
\code{SAGE_ROOT/doc/commontex/macros.tex}.

\item Use \code{view(x)} to view the typeset version of an object x.
\end{enumerate}
%skip
An example template for \Latex representation follows:
+An example template for a \code{_latex_} method follows:
\begin{verbatim}
import sage.misc.latex as latex
...
class X:
...
def _latex_(self):
@@ 540,19 +691,26 @@
return `\\frac{%s}{%s}''%(latex(self.numer), latex(self.denom))
\end{verbatim}
+As shown in the example, \code{latex(a)} will produce \Latex code
+representing the object \code{a}. Calling \code{view(a)} will display the
+typeset version of this.
+
+
+
\subsection{Print Representation}
The standard Python printing method is \code{__repr__(self)}. In
\sage, that is for objects that derive from \code{SageObject} (which
is everything in \sage), instead define \code{_repr_(self)}. This is
preferable because if you only define \code{_repr_(self)} and not
\code{__repr__(self)}, then users can rename your object to print
however they like.
+is everything in \sage), instead define \code{_repr_(self)}.
+% This is preferable because if you only define \code{_repr_(self)} and
+% not \code{__repr__(self)}, then users can rename your object to print
+% however they like.
Here is an example of the \code{_latex_} and \code{_repr_} functions
for the Pi class. It is from the file
\code{SAGE_ROOT/devel/sage/sage/functions/constants.py}
+\code{SAGE_ROOT/devel/sage/sage/functions/constants.py}:
\begin{verbatim}%skip
+%skip
+\begin{verbatim}
class Pi(Constant):
"""
The ratio of a circle's circumference to its diameter.
@@ 571,35 +729,38 @@
return "\\pi"
\end{verbatim}
\subsection{Matrix from Object}
Provide a \code{_matrix_} member function for an object that can be
+\subsection{Matrix or Vector from Object}
+Provide a \code{_matrix_} method for an object that can be
coerced to a matrix over a ring $R$. Then the \sage function
\code{matrix} will work for this object.
The following is from \code{SAGE_ROOT/devel/sage/sage/graphs/graph.py}
+The following is from \code{SAGE_ROOT/devel/sage/sage/graphs/graph.py}:
\begin{verbatim}
class SimpleGraph(GenericGraph):
 ...
 def _matrix_(self, R):
 return self.am()
+class GenericGraph(SageObject):
+ ...
+ def _matrix_(self, R=None):
+ if R is None:
+ return self.am()
+ else:
+ return self.am().change_ring(R)
 def am(self):
 """
 Shorter call for adjacency matrix makes life easier.
 """
 return self.adjacency_matrix()
+
+ def adjacency_matrix(self, sparse=None, boundary_first=False):
+ ...
+
+ am = adjacency_matrix # shorter call makes life easier
\end{verbatim}
\subsection{Vector from Object}
Provide a \code{_vector_} member function for an object that can be
+Similarly,
+provide a \code{_vector_} method for an object that can be
coerced to a vector over a ring $R$. Then the \sage function
\code{vector} will be work for this object.
+\code{vector} will work for this object.
\todo{Provide example from a .py file
The following is from the file
 SAGE_ROOT/sage/sage/modules/free_module_element.pyx
+ \code{SAGE_ROOT/sage/sage/modules/free_module_element.pyx}:
\begin{verbatim}
cdef class FreeModuleElement(element_Vector): # abstract base class
@@ 610,15 +771,92 @@
}
+\section{\SAGE Preparsing}
+\label{sec:Preparsing}
+
+The following files are relevant to preparsing in \sage:
+
+\begin{enumerate}
+
+\item \code{SAGE_ROOT/local/bin/sagesage},
+
+\item \code{SAGE_ROOT/local/bin/sagepreparse},
+
+\item \code{SAGE_ROOT/devel/sage/sage/misc/preparser.py}
+
+% \item \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py}
+
+\todo{Talk about
+ \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py} file}
+
+\end{enumerate}
+
+In particular, the file \code{preparser.py} contains the \SAGE
+preparser code. Here are some , and the following are some notes from it:
+
+\begin{itemize}
+
+\item In \SAGE, methods can be called on integer and real literals 
+ note that in pure Python this would be a syntax error. For example:
+\begin{verbatim}
+ sage: 16.sqrt()
+ 4
+ sage: 87.factor()
+ 3 * 29
+\end{verbatim}
+
+\item Raw literals are not preparsed, which can be useful from an
+ efficiency point of view. Just like Python ints are denoted by an
+ L, in \SAGE raw integer and floating literals are followed by an ``r''
+ (or ``R'') for raw, meaning not preparsed. Example:
+\begin{verbatim}
+ sage: a = 393939r
+ sage: a
+ 393939
+ sage: type(a)
+
+ sage: b = 393939
+ sage: type(b)
+
+ sage: a == b
+ True
+\end{verbatim}
+
+\item Raw literals can be very useful in certain cases. For instance,
+ Python integers are typically much more efficient than \SAGE integers
+ when they are very small; large \SAGE integers are much more
+ efficient than Python integers, since they are implemented using the
+ GMP C library. For example, the first of these commands may be somewhat
+ slower than the second:
+\begin{verbatim}
+ sage: v = [ 2*i for i in range(10000)]
+ sage: v = [ 2r*i for i in range(10000r)]
+\end{verbatim}
+because the first uses \sage integers, while the second uses raw
+Python integers.
+
+\end{itemize}
+
+Consult the file \code{preparser.py} for more details about \SAGE
+preparsing, more examples involving raw literals, etc.
+
+When a file \code{foo.sage} is loaded in a \sage session, a preparsed version of
+\code{foo.sage} is created titled \code{foo.py}. The beginning of \code{foo.py} states:
+\verb+This file was *autogenerated* from the file foo.sage.+
+
+
\section{The \SAGE Rules and Conventions for Coercion and Arithmetic}
\note{These are the {\bf planned} rules and conventions. They have not
been systematically implemented throughout the system yet.}
Let $R$ and $S$ be \SAGE parent structures, e.g.,
groups, rings, point sets, etc.\todo{Make section gentler}
+\note{August 2008: Much of this material is out of date. We are
+working on a revised version.}
+
+\todo{Make section gentler}
+
+Let $R$ and $S$ be \SAGE parent structures, e.g., groups, rings, point
+sets, etc.
\begin{itemize}
\item {\bf OBJECT COERCION \code{_coerce_}:}
+\item \textbf{OBJECT COERCION \code{_coerce_}:}
Suppose a \code{_coerce_} map $R \to S$ is defined. This
is a method of $S$, so it is called using the notation \code{S._coerce_(x)}, where $R$ is a parent of $x$.
It's signature is
@@ 679,29 +917,29 @@
rules. E.g., embeddings of finite fields via Conway polynomials, or
inclusions of extensions of number fields, fit into this structure, as
does the inclusion $\QQ \hookrightarrow \CC$ and the surjection $\ZZ
\to \ZZ/n\ZZ$. The function \code{_coerce_} {\bf does not} have to be
+\to \ZZ/n\ZZ$. The function \code{_coerce_} \textbf{does not} have to be
``canonical'' in a precisely defined mathematical sense.
\item {\bf OBJECT CREATION \code{__call__}:}
+\item \textbf{OBJECT CREATION \code{__call__}:}
If you write $R(x)$, then the \code{__call__} method of $x$ is called. It
should make an object of $R$ from $x$, if at all possible.
\code{__call__} is never called implicitly by binary operators.
If $x$ already has $R$ as its parent, then \code{__call__} must return
 a {\bf new copy} of $x$, unless $x$ is immutable (this is to agree with
+ a \textbf{new copy} of $x$, unless $x$ is immutable (this is to agree with
standard Python conventions).
For example, if $x \in \ZZ/n\ZZ$, then \code{ZZ.__call__(x)} is defined but
\code{ZZ.__coerce__(x)} is not.
\item {\bf ARITHMETIC \code{__add__}, \code{__mul__}, ...:}: When
+\item \textbf{ARITHMETIC \code{__add__}, \code{__mul__}, ...:}: When
doing a binary operation, if the parents are not identical (in the
sense of is\todo{incomplete?}), determine if precisely one {\tt
\_coerce\_} map is defined; if so, apply it and do the arithmetic
operation. If both are defined, the parents are canonically
isomorphic, so use the left one. If neither are defined, raise a
 TypeError. (Whether or not there is a coerce map between objects \todo{the objects?}
+ TypeError. (Whether or not there is a coerce map between objects
should be cached for efficiency.)
\item {\bf PARENT OBJECT \code{__cmp__}}: \code{R == S} by definition
+\item \textbf{PARENT OBJECT \code{__cmp__}}: \code{R == S} by definition
means that \code{_coerce_} is defined in both directions. Roughly
speaking this means that $R$ and $S$ are identical or canonically
isomorphic, though there could be some exceptions to this. Note
@@ 709,51 +947,42 @@
considered equal, since there are never canonical coercion maps in
both directions when the categories are different.
\item {\bf ELEMENT \code{__cmp__}}:
+\item \textbf{ELEMENT \code{__cmp__}}:
If the parents aren't identical, test if
 precisely one \code{_coerce_} map is defined  if so, return \code{__cmp__}
+ precisely one \code{_coerce_} map is defined  if so, return \code{__cmp__}
after applying the coerce. If both coercions are defined, compute
both \code{__cmp__}'s (in both $R$ and $S$). Return the value if they give
the same results. Otherwise return $1$ (the convention in Python
is that \code{__cmp__} never raises an exception). If no \code{_coerce_} is
defined return $1$ (i.e., not equal).
\item {\bf IN \code{__contains__}}:
+\item \textbf{IN \code{__contains__}}:
\code{x in R} should be true if and only if \code{R._coerce_(x)} would not
raise a \code{TypeError}.
\end{itemize}
+
\section{Mutability}
+
Parent structures (e.g., rings, fields, matrix spaces, etc.) should
be immutable and globally unique whenever possible. Immutability
includes making it so properties like generator labels and default
+means, among other things, that properties like generator labels and default
coercion precision cannot be changed.
Global uniqueness while not wasting space storing objects that are not
being referenced is best implemented using the standard Python weakref
+Global uniqueness while not wasting memory
+is best implemented using the standard Python weakref
module, a factory function, and module scope
variable.\todo{Rewrite. Difficult to parse. Make gentler}
+variable.
+\todo{Rewrite. Difficult to parse. Make gentler}
\todo{Put a tutorial on this here}.
+\todo{Put a tutorial on this here}
Certain objects, e.g., matrices, may start out mutable and become
immutable later. See the file \code{SAGE_ROOT/devel/sage/sage/structure/mutability.py}.
\section{Importing Modules}
Do not import at the top level of your module a thirdparty module that
will take a long time to initialize (e.g., matplotlib). It is important
that \code{from sage.all import *} not take forever, since this is what
dominates the \SAGE startup time.

On a somewhat regular basis I do the following:
\begin{verbatim}
 echo "%prun import sage.all"  sage ipythonmore
\end{verbatim}
in order to profile what is taking a long time during
the load. In many cases I use the locate command to
find files mentioned in the profile.
\section{The {\tt \_\_hash\_\_} Special Method}
+
Here's the definition of \code{__hash__}
from the Python reference manual.
@@ 780,7 +1009,7 @@
Python language, {\em which in \sage we simply cannot make} (!), and
violating it has consequences. Fortunately, the consequences are pretty
clearly defined and reasonably easy to understand, so if you know
about them they don't cause you trouble. I think the following
+about them they don't cause you trouble. The following
example illustrates them pretty well:
\begin{verbatim}
sage: v = [Mod(2,7)]
@@ 800,7 +1029,7 @@
KeyError: 9
\end{verbatim}
Here's another example,
+Here's another example:
\begin{verbatim}
sage: R = RealField(10000)
sage: a = R(1) + R(10)^100
@@ 811,7 +1040,7 @@
Unfortunately, in \SAGE we simply can't require:
\begin{verbatim}
 (*) "a == b ==> hash(a) == hash(b)"
+ (*) "a == b ==> hash(a) == hash(b)"
\end{verbatim}
because serious mathematics is simply too complicated for this
rule. For example, the equalities
@@ 820,771 +1049,149 @@
integers.
The only way we could ``fix'' this problem for good would be to
abandon using the "==" operator for "\SAGE equality", and implement
+abandon using the \code{==} operator for ``\SAGE equality'', and implement
\SAGE equality as a new method attached to each object. Then we could
follow Python rules for "==" and our rules for everything else, and
+follow Python rules for \code{==} and our rules for everything else, and
all \SAGE code would become completely unreadable (and for that matter
unwritable). So we just have to live with it.
So what is done in \sage is to {\em attempt} to satisfy (*) when it
+So what is done in \sage is to {\em attempt} to satisfy \verb+(*)+ when it
is reasonably easy to do so, but use judgment and not go overboard.
For example,
\begin{verbatim}
sage: hash(Mod(2,7))
2
\end{verbatim}
I.e., we get 2. That's better than some weird random hash that also
+The output 2 is better than some random hash that also
involves the moduli, but it's of course not right from the Python point
of view, since \code{9 == Mod(2,7)}.
The goal is to make a hash function that is fast but within reason
respects any obvious, natural inclusions and coercions.
+respects any obvious natural inclusions and coercions.
+\section{Exceptions}


\chapter{Creating a \SAGE Module or Package}

\todo{What is a \SAGE Module and what is a \SAGE Package? Are they the
 same? This chapter talks about only the latter? When
 should/shouldn't one make a package out of a bunch of files?}

\begin{enumerate}
\item Design: Think about what your program will do and how that fits into the
 structure of \SAGE. In particular, much of \SAGE is implemented in
 the objectoriented language Python, and there is a hierarchy of
 classes that organize code and functionality. It is {\em
 surprisingly painful and difficult} to create some Python classes,
 then realize that much functionality should be factored out into
 base classes, and to do it\todo{Not sure what you are trying to
 convey here}. So try to get the structure right the first time.
 For example, if you implement elements of a ring your class should
 derive from
 \class{sage.structure.element.RingElement}.\todo{Design is being
 talked about without using the word. Consciously avoiding the
 term?}

\item Coding: Code it up, following the conventions in
 Chapter~\ref{ch:conventions}. Make the basic structure and
 interface for what you're doing in Python. You can use any of the
 following languages to implement the hard parts: Python, C/C++,
 Cython, Fortran 95, GAP, Singular, and GP/PARI. You can also use the mwrank,
 GSL, NTL, and PARI C/C++ libraries. (And if you are OK with your code
 depending on optional \sage packages, you can use Octave, or
 even Magma, Mathematica, or Maple.) For readability and usability
 it is best to use \sage when possible. On the other hand, for
 efficiency reasons it is often crucial to use a compiled language or
 an existing library or programdo not reinvent the wheel or write
 code that is so slow as to be only a toy\todo{I split the
 original line into two, as reusing existing code should be
 emphasized more}.

\item Documentation: Document {\em every} Python function that you've
 defined,

\item Testing: Liberally include examples in your source code. See
 Chapter~\ref{ch:testing} for more on automated testing of examples.

\item (Optional) Write a file that generates random input to test (and
 break!) your code with the intention of making it robust (See
 Chapter~\ref{ch:randomtesting}).

\item (Optional) Send your package to the \code{sagedevel} Google group
 for inclusion in \SAGE, or make a standalone
 {\tt spkg} (see Chapter~\ref{ch:spkg}).

\end{enumerate}


\section{\SAGE Preparsing}
\label{sec:Preparsing}

The following is a list of files relevant to preparsing through which
control flows when \sage is run:

\begin{enumerate}

\item \code{SAGE_ROOT/sagesage},

\item \code{SAGE_ROOT/sagepreparse},

\item \code{SAGE_ROOT/devel/sage/sage/misc/preparser.py}

 \todo{Talk about
 \code{SAGE_ROOT/devel/sage/sage/misc/preparser_ipython.py} file}

\end{enumerate}

The file preparser.py is contains the \SAGE preparser code and the
following are some notes from the aforementioned file

\begin{itemize}

\item In \SAGE methods can also be called on integer and real literals
 (note that in pure Python this would be a syntax error).

\item Note that calling methods on int literals in pure Python is a
 syntax error, but \SAGE allows this for \SAGE integers and reals,
 because users frequently request it.

\item Raw literals are not preparsed, which can be useful from an
 efficiency point of view. Just like Python ints are denoted by an
 L, in \SAGE raw integer and floating literals are followed by an ``r''
 (or ``R'') for raw, meaning not preparsed.

\item Raw literals can be very useful in certain cases. For instance,
 Python integers are typically much more efficient than \SAGE integers
 when they are very small; large \SAGE integers are much more
 efficient than Python integers, since they are implemented using the
 GMP C library  see section \ref{sec:Benchmarking}.

\end{itemize}


Consult preparser.py for details about \SAGE preparsing, examples involving raw literals, etc.

When foo.sage is loaded in a sage session, a preparsed version of
foo.sage is created titled foo.py. The beginning of foo.py states:
\code{This file was *autogenerated* from the file foo.sage.}

\section{Randomized Testing}
\label{ch:randomtesting}
In addition to all the examples, which serve as both demonstrations
and tests of your code, you should also create a test suite. Think of
this as a program that will run for a while and ``tries'' to crash
your code using randomly generated input. Your test code should
define a class \class{Test} with a {\tt random()} method that runs
random tests. These are all assembled together later, and each test
is run for a certain amount of time on a regular basis.

\section{Creating a New \sage spkg Package}
\label{ch:spkg}
Here's how to make your own \sage package.

\begin{itemize}
\item Note that a \sage package is simply a .tar.bz2 file (or a tar
 file), but named with .spkg to discourage confusion. In particular,
 you can type
+Please avoid code like this:
%skip
\begin{verbatim}
 tar jxvf my_packageversion.spkg
+try:
+ some code
+except: # bad
+ more code
\end{verbatim}
to extract one and see what is inside.
+Instead catch specific exceptions. For example,
+\begin{verbatim}
+try:
+ return self.__coordinate_ring
+except (AttributeError, other exceptions), msg: # Good
+ more code to compute something
+\end{verbatim}
+Note that the syntax in \code{except} is to list all the exceptions
+that are caught as a tuple, followed by an error message.
\item
To create your own package:
\begin{itemize}
\item[(a)]
Make a directory, e.g., \code{mypackage0.1}.
The name of the directory should be a lowercase string with no
dashes followed by a dash followed by a version number.
+If you don't have any exceptions explicitly listed (as a tuple), your
+code will catch absolutely anything, including \code{ctrlC} and alarms,
+and this will lead to confusion.
\item[(b)]
Put your files in that directory.
\item[(c)]
Create an executable shell script \code{mypackage0.1/spkginstall}.
+\section{Importing}
\item[(d)]
The script \code{spkginstall} is run during installation
of the \sage package. In this script you may
make the following assumptions:
\begin{itemize}
\item
The PATH has \code{sage} and the \sage install's \code{python}
location at the front. Thus the command
+We mention two issues with importing: circular imports and importing
+large thirdparty modules.
+
+First, you must avoid circular imports. For example, suppose that the
+file \code{SAGE_ROOT/devel/sage/sage/algebras/steenrod_algebra.py}
+started with a line
%skip
\begin{verbatim}
 python setup.py install
+from sage.sage.algebras.steenrod_algebra_bases import *
\end{verbatim}
will run the correct version of Python with
everything setup correctly. Also, e.g.,
running \code{gap} or \code{Singular} will run the
correct versions.

\item
The environment variable \code{SAGE_ROOT} points to
the root directory of the \sage install.
\item
The environment variable \code{SAGE_LOCAL} points to
the \code{SAGE_ROOT/local} directory of the \sage install.
\item
The environment variables
\code{LD_LIBRARY_PATH} and \code{DYLD_LIBRARY_PATH}
both have \code{SAGE_ROOT/local/lib} at the front.

\end{itemize}

\item[(e)]
The \code{spkginstall} script should copy your files to the appropriate
place after doing any build that is necessary. That's it!

\item[(f)] (Optional) Send a copy to the \code{sagedevel} Google group
so it can be posted to the \SAGE web site, so anybody can automatically
install it by typing \code{sage i my_packageversion.spkg}.
\end{itemize}
\end{itemize}

For example, the location of GAP is \code{SAGE_ROOT/local/lib/gap4.4.6/}.
Thus to install gap code you have to copy it there. You should check
that the directory with the given version your package is created for
exists in your \code{spkginstall} script, e.g., with the following code:
+and that the
+file \code{SAGE_ROOT/devel/sage/sage/algebras/steenrod_algebra_bases.py}
+started with a line
%skip
\begin{verbatim}
if [ ! d $SAGE_ROOT/local/lib/gap4.4.6 ]; then
 echo "This package requires that GAP 4.4.6 is installed."
 exit 1
fi
+from sage.sage.algebras.steenrod_algebra import SteenrodAlgebra
+\end{verbatim}
+This sets up a loop: loading one of these files requires the other,
+which then requires the first, etc.
+
+With this setup, running \sage will produce an error:
+%skip
+\begin{verbatim}
+Exception exceptions.ImportError: 'cannot import name SteenrodAlgebra' in 'sage.rings.polynomial.polynomial_element.Polynomial_generic_dense.__normalize' ignored
+
+ImportError Traceback (most recent call last)
+
+...
+ImportError: cannot import name SteenrodAlgebra
+\end{verbatim}
+Instead, you might replace the \code{import *} line at the top of the
+file by more specific imports where they are needed in the code
+ e.g., the \code{basis} method for the class \code{SteenrodAlgebra}
+might look like this (omitting the documentation string):
+%skip
+\begin{verbatim}
+ def basis(self, n):
+ from steenrod_algebra_bases import steenrod_algebra_basis
+ return steenrod_algebra_basis(n, basis=self._basis_name, p=self.prime)
\end{verbatim}
\emph{Caveat}: Do {\em not} just
copy to \code{SAGE_ROOT/local/lib/gap*/} since that will copy
your package to the lib directory of the {\em old} version of GAP if
GAP is upgraded.
+Second, do not import at the top level of your module a thirdparty
+module that will take a long time to initialize (e.g., matplotlib).
+As above, you might instead import specific components of the module
+when they are needed, rather than at the top level of your file.
External Magma code goes in
\verb+SAGE_ROOT/data/extcode/magma/user+, so if you want to redistribute
Magma code with \sage as a package that Magmaenabled users can use,
that's where you'd put it. You'd also want to have relevant Python code
to make the Magma code easily usable.
+It is important to try to make \code{from sage.all import *} as fast
+as possible, since this is what dominates the \SAGE startup time, and
+controlling the toplevel imports helps to do this.
{\bf Example}:
These instructions provide directions for creating a particular
\sage package. \todo{Something about this example bugs me and I'm not sure what it is.}
\begin{itemize}
\item
Download \code{guava2.4.tar.gz} (the tarball for GUAVA, the
GAP errorcorrecting codes package) from
\code{http://www.gapsystem.org/Packages/guava.html}.
+\section{Using Optional Packages}
\item
Suppose you have a directory \code{sagepackageguava}.
Within that, create a directory \code{guava2.4}.
Extract the tarball into that directory
(so there will be a \code{guava2.4} subdirectory of
\code{sagepackageguava}).
+If a function requires an optional package, that function should fail gracefully 
+perhaps using a \code{try} and \code{except} block  when the optional
+package is not available, and should give a hint about how to install
+it. For example, typing \code{sage optional} gives a list of all
+optional packages, so it might suggest to the user that they type that.
\item
Put the following script called \code{spkginstall} in
\code{sagepackageguava}:
%skip
\begin{verbatim}
#!/bin/sh
if [ ! d $SAGE_ROOT/local/lib/gap4.4.6 ]; then
 echo "This package requires that GAP 4.4.6 is installed."
 exit 1
fi
+\chapter{Coding in Other Languages}
cp pr guava2.4 $SAGE_ROOT/local/lib/gap4.4.6/pkg/
+When writing code for \sage, use Python for the basic structure and
+interface. For speed, efficiency, or convenience, you can implement
+parts of the code using any of the following languages:
+Cython, C/C++, Fortran 95, GAP, Singular, and GP/PARI. You can also
+use the mwrank, GSL, NTL, and PARI C/C++ libraries. (And if you are
+okay with your code depending on optional \sage packages, you can use
+Octave, or even Magma, Mathematica, or Maple.)
cd $SAGE_ROOT/local/lib/gap4.4.6/pkg/guava2.4
./configure ../..
make
\end{verbatim}
+The first section of this chapter discusses Cython (which is a
+compiled language, based on Python); many components of \sage are
+written in Cython. Later sections discuss the interfaces between
+\sage and PARI, GAP, and Singular.
Make it executable with \code{chmod +x spkginstall}.
\item
In the directory \code{sagefiles},
issue the command
%skip
\begin{verbatim}
sage pkg guava2.4
\end{verbatim}
where \verb++ means the absolute path to the sage
script. This will do some checks, run tar/bz2, and create the spkg file,
\code{guava2.4.spkg}, in the
directory \code{sagefiles},
\end{itemize}
+\section{Cython}
To test this, within the \code{sagefiles} directory
(or whatever directory contains \code{guava2.4.spkg}),
type
+Cython is a compiled version of Python; it is based on Pyrex
+(\url{http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/}). To
+a large degree, Cython has changed based on what \sage's developers
+needed: Cython has been developed in concert with \sage.
%skip
\begin{verbatim}
sage i guava2.4.spkg
\end{verbatim}
It should install and compile the guava program in sage.
To test its functionality, one could do the following:
+As such, it is a young, but developing, language, with young, but
+developing, documentation. See its web page,
+\url{http://www.cython.org/}, for the most uptodate information.
\begin{verbatim}
sage: gap(`LoadPackage("guava")') # requires optional package
true
sage: gap(`HammingCode(3,3)') # requires optional package
a linear [13,10,3]1 Hamming (3,3) code over GF(3)
\end{verbatim}

To build again if you've already installed the package, use
%skip
\begin{verbatim}
sage f guava2.4.spkg
\end{verbatim}


\chapter{Documentation}

\section{Source Code Documentation}
Use \Latex formatting in the documentation string. If you wish to use
backslashes in the documentation string, use double backslashes or place an {\tt r} right before the first triple opening quote. For example,
%skip
\begin{verbatim}
def cos(x):
 """
 Returns $\\cos(x)$.
 """

def sin(x):
 r"""
 Returns $\sin(x)$.
 """
\end{verbatim}

The nonstandard macros that are used when typesetting the
documentation are defined in
\code{SAGE_ROOT/doc/commontex/macros.tex}. If you need to add more macros,
add them there and post them on the \code{sagedevel} Google group so that
they can be included in the next version of \sage.


The first line two or three lines of the docstring should briefly summarize
what the function does, e.g., ``Returns X'' or ``Computes X'',
or ``Creates a new X''. Next describe the input and output of
the function using the following notation:
%skip
\begin{verbatim}
INPUT:
 var1  (default: ...) description
 var2  (default: ...) description
 etc.

OUTPUT:
  description of return value 1
  description of return value 2
 etc.
\end{verbatim}

This is a placeholder of a "Perfect example of the above".\todo{
I=ifti propose everywhere in this document when there is a template, it is followed by a path/filename where in one can find a "perfect" example.}

For consistency you should use the above format, which is used
throughout the system. Do not just describe the input and output
in long sentences. For methods of a class, whether or not
to describe self is up to you; this can be useful if self must
have some special properties. Also, you do not have to
specify the types of the input and output arguments
precisely; e.g., use ``integer'' for an int or long or
\sage \code{Integer}.

Every function should have at least one, and preferably many, examples
that illustrate its usage. These examples must exactly match the
output of an actual running copy of Python. To test whether this is
the case, run the program \code{sage t} with your Python or Cython
file as unique argument (see Chapter~\ref{ch:testing} for more details).

For more details, see Section~\ref{sec:convdocs}.\todo{redundancy \S 2.4 and \S4.1}

\section{Extra Documentation}
You might also write additional documentation in \Latex, which is not to
be part of any source code file. The examples in this documentation
should be contained in verbatim environments. The examples should be
tested using \code{sagetesttex}, which you run on the \Latex file. Put
\%skip on the line preceding the environment to skip testing of an
example.

Note that \code{sagetesttex} is completely different than \code{sagedoctest}. It extracts the documentation and feeds it to a
running instance of \SAGE.


\section{Automated Testing}\label{ch:testing}
This section describes automated testing of test files of the
following types: \code{.py .pyx .sage .tex}. Use \code{sage t
 } to test that the examples in \code{} behave exactly as
claimed. See Section~\ref{sec:convdocs} for a discussion of how to
include examples in documentation strings and what conventions to
follow  Section ~\ref{sec:Furtherconventions}.

\subsection{Testing .py, .pyx and .sage Files}\label{sec:test:python}
Run \code{sage t } to test that all code examples in
\code{filename.py}. Similar remarks apply to \code{.sage} and
\code{.pyx} files.

\begin{verbatim}
 sage t [verbose] [optional] [files and directories ... ]
\end{verbatim}

When you run \code{sage t }, \sage makes a copy of
\code{} with all the \code{sage} prompts replaced by
\code{{>}{>}>}, then uses the standard Python doctest framework to test
the documentation. More precisely,
the Python script \code{local/bin/sagedoctest} implements
documentation testing. It does the following when asked
to test a file \code{foo.py} or \code{foo.sage}.
\begin{enumerate}

\item Creates the directory \code{.doctest} if it doesn't
exist and the file \code{.doctest/foo.py}.

\item The file \code{.doctest_foo.py} contains functions for
each docstring in \code{foo.py}, but with all \sage preparsing
applied and with \code{from sage.all import *} at the top.
The function documentation is thus standard Python with
\code{{>}{>}>} prompts.

\item \code{sagedoctest} then runs \sage's Python interpreter
on \code{.doctest_foo.py}.
\end{enumerate}

\note{{\em Note that line numbers in the errors
you might see apply to the file \code{.doctest_foo.py}, not
to the original file \code{foo.py}!} If you get errors,
feel free to inspect \code{.doctest_foo.py}; it is never
automatically deleted by \sage.}

Your file passes these tests if the code in it will run when entered
at the \code{sage:} prompt with no special imports. Thus users are
guaranteed to be able to exactly copy code out of the examples you
write for the documentation and have them work.

%skip
\begin{verbatim}

William:

I changed the doctest behavior recently so that now you can doctest
files anywhere (not necessary in \SAGE), and type e.g.,

 sage: from foo import *
 sage: ...

in the doctest blocks and it will work, even if foo.py isn't part of the
\SAGE library. It's much better now. Feel free to change the reference
manual to reflect this.

\end{verbatim}


\subsection{Testing \Latex Documentation}

Run \code{sage t } to test the examples in
verbatim environments in \Latex documentation. Put
\code{\%skip}
on the line right before a verbatim environment to have that
example skipped when testing the file.

\sage creates a file \code{.doctest_filename.py} and tests
it just as for \code{.py}, \code{.pyx} and \code{.sage}
files. In order to skip testing of a block of code
in a verbatim environment, put the \Latex comment
\code{\%skip} on the previous line.

One difference is that in \Latex files one
often inserts explanatory texts between different
verbatim environments. To link together verbatim
environments use the \code{\%link} comment. For
example\todo{I have changed verbatim to verbatim0 here
since I don't know how to include end{verbatim} in a
verbatim environment.}
%skip
\begin{verbatim}
 \begin{verbatim}
 sage: a = 1
 \end{verbatim_}%link

 Next we add 1 to \code{a}.
 %link
 \begin{verbatim}
 sage: 1 + a
 2
 \end{verbatim_}
\end{verbatim}

See \code{SAGE_ROOT/doc/tut/tut.tex} for many examples
of how to include automated testing in \Latex documentation
for \sage.



\section{Modifying the \SAGE Manuals}
We use the tutorial as an example.

You just modify the file \code{SAGE_ROOT/devel/doc/tut/tut.tex} To
build a test copy of the tutorial with your changes, type
\code{./build_dvi} in the \code{devel/doc/tut} directory. You'll get
a file \code{tut.dvi}.
You can also build HTML versions of everything by typing \code{make
 html} in the \code{devel/doc} directory. Build pdfs with \code{make
 pdf}. \todo{You could also do \code{./build_pdf} to create a pdf directly from the tex file.}

It is a good idea to type \code{sage t tut.tex} to make sure any
examples you added to the tutorial work as claimed.

The \code{SAGE_ROOT/devel/doc} directory is a Mercurial repository
(see Chapter~\ref{ch:mercurial}), so you can see exactly what changes
you've made to the documentation, make patches of what you've done
available, etc. You can do this from within \sage by typing
\code{hg_doc.[tab]}.

Imagine you have edited this Programming Manual found in \code{devel/doc/prog}.
Before you started you did a pull to be sure your docs are up to date:
\todo{Give some examples of usage}

%skip
\begin{verbatim}
sage: hg_doc.pull()
cd "/home/jaap/sage/devel/doc" && hg status
cd "/home/jaap/sage/devel/doc" && hg status
cd "/home/jaap/sage/devel/doc" && hg pull u http://sagemath.org/sage//hg//docmain
pulling from http://sagemath.org/sage//hg//docmain
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 1 files
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
If it says use 'hg merge' above, then you should
type hg_sage.merge(), where hg_sage is the name
of the repository you are using. This might not
work with the notebook yet.
\end{verbatim}

After you made many changes, you want to commit them:
%skip
\begin{verbatim}
sage: hg_doc.commit()
cd "/home/jaap/downloads/sage1.6/devel/doc" && hg diff  less
cd "/home/jaap/downloads/sage1.6/devel/doc" && hg commit
\end{verbatim}

You saw the differences piped through \code{diff}, you were asked
to comment: type 'i' or 'a', your comment, hit 'Escape' and type ':wq".

Now you can make a patch 'patchname'. e.g.:
%skip
\begin{verbatim}
sage: hg_doc.send('prog20070118.hg')
Writing to /home/jaap/sage/prog20070118.hg
cd "/home/jaap/sage/devel/doc" && hg bundle tmphg http://sagemath.org/sage//hg//docmain
searching for changes
Successfully created hg patch bundle /home/jaap/sage/prog20070118.hg
\end{verbatim}

Send the file 'patchname' to the \code{sagedevel} Google group.


\chapter{Writing Optimized Compiled Code With \SAGE}
This chapter is about how to write fast compiled optimized code for
\sage. It was written mainly by William Stein with contributions
from David Harvey, Martin Albrecht, David Joyner, Alex Clemesha.
Some of the text is closely copied or translated (to mathematical
language) from Greg Ewing's original Pyrex manual.

This chapter assumes that you have a C compiler suite (e.g., GCC)
installed on your computer, since you should not write compiled
extension code on a computer without a C compiler. If you are using
OS X, download and install the latest version of XCode.\footnote{The
 download is about 1GB, and you must sign up for the Apple Developer
 Network. \url{http://developer.apple.com/tools/xcode/}}
For Linux, use
your package manager to install GCC. If you're using Microsoft
Windows, then you should be using \sage via colinux, which includes
GCC by default.


%[[General discussion of when to use compiled code versus interpreted
%code.]]

Interpreted code: way easier to debug,
trace, document, structure (multiple inheritence), etc.;
good when few for loops; lots of structure and calls
to compiled code.

Compiled code: two reasons to usemake fast code and
make optimal use of libraries written in a compiled language;
hard to write and debug; takes longer to
write. The methods explained in this chapter can produce
code where speed is only limited by your choice of algorithm.



{\bf TODO:}
\begin{enumerate}
\item Module name and file name must agree!
%[(Check this at build time?)]

\end{enumerate}

\section{Introduction}
This chapter describes how to write \SAGE programs using Cython, which
is a language for writing Python extension modules that is a very
slightly variant of Pyrex (we will often refer to it as ``Pyrex''
still).

The primary goal of this chapter is to provide excellent documentation
about how to use Cython to write very fast code in the context of
\SAGE. (Currently it is a mess.) This chapter is self contained in
that the reader should not have to look at any of the documentation
distributed with Pyrex. The reader is urged to at some point read the
Python/C API Reference manual.

There is also much to be learned from the Pyrex
mailing list:

\url{http://lists.copyleft.no/mailman/listinfo/pyrex}.

{\bf Acknowledgement:} The existence of Greg Ewing's Pyrex is
one of the primary reasons Stein chose Python as the implementation
language for \SAGE. It is a brilliantly useful tool.

%\cite{pyrex:list}.

%A list of quality open source projects that
%use Pyrex: soya3d, ... [[add more here]]


\subsection{Interpreted and Compiled Code}
Every serious computer algebra system, e.g., MAGMA, PARI, Mathematica,
Maple, GAP, Singular, etc., is implemented as a combination of
compiled and interpreted code. For Mathematica, Singular, MAGMA, and
PARI, most of the implementation is in compiled C (or C++) code; some
of these systems tend to be very optimized (subject to the constraints
of the algorithms they implement). In contrast, Maple and GAP have a
relatively small compiled ``kernel'' that defines the underlying
programming language; most of the system is then implemented in this
language. If you do benchmarks you'll discover that Mathematica is
{\em much} faster than Maple at some basic operations. Likewise,
benchmarks reveal that MAGMA is often faster than GAP.
%[[give
%examples that illustrate each of these points. E.g., basic
%arithmetic where maple is pitiful, computation of Bernoulli
%numbers where PARI crushes all others.]]

This fusion of interpreted and compiled code is extremely natural for
mathematics software; some algorithms are much better implemented in
an interpreter because all timecritical steps involve low level
arithmetic  other algorithms, e.g., matrix multiplication, must be
implemented in a compiled language in order to perform optimally.
Also, existing compiled libraries are sometimes of very high quality,
and a compiled language is needed to create the best possible
interface to such libraries. It's crucial that both approaches to
programming be fully supported. When deciding how to implement \SAGE,
I searched for months for an environment that could support both
approaches. Python and Cython provide exactly this in a way that I
believe is much easier conceptually than the implementation models of
any of the other systems mentioned above.

\subsection{Why Cython is the Best Available Option for \SAGE}
The software listed in this section is used either
to speed up Python programs or to make code written in
C++, C, FORTRAN, etc., available in Python. This section briefly
explains what each tool does and why it is {\em not} included
in \SAGE or used in the implementation of the \SAGE library.


\subsubsection{SWIG}
SWIG (Standard Wrapper and Interface Generator)
takes C++ (and C, etc.) libraries, and generates
wrapper code so that the code can be used from
within Python (and in several other interpreters). One
writes interface files that describe how to make Python
objects usable from your C++ code, and conversely what
transformations C++ objects should undergo to be usable from Python.

In 2004, Stein considered using SWIG for \SAGE extensively, and implemented
several basic types using this model. The idea was to
write code in C++ for \SAGE that needed to be fast, then wrap it in
SWIG. This ground to a halt, because the result was not
 {\em sufficiently} fast. First, there is overhead when writing code
in C++ in the first place. Second, SWIG generates several layers of
code between Python and the code that does the actual work; for
implementing any of \SAGE's numerous basic arithmetic types,
e.g., integers, the overhead is completely unacceptable, at
least if one wants code that is as fast as Magma and PARI. For
example, in my initial implementation of integer arithmetic using
SWIG and GMP, there were the following {\em six} levels between the
\sage interpreter and the GMP C library:
\begin{center}
\begin{tabular}{l}\hline
Python code to provide a clean interface\\\hline
SWIG Autogenerated Python code \\\hline
SWIG Autogenerated C++ extension code \\\hline
Handcode C++ Integer class\\\hline
GMP's C++ Interface\\\hline
GMP C Library\\\hline
\end{tabular}
\end{center}
In this model, just multiplying two integers is extremely complicated,
and slow. Moreover, writing the SWIG interface files was
painful, since general Python types are difficult to work with from
C++ code. In contrast, with Cython the situation is as follows
(see {\tt integer.pyx}):
\begin{center}
\begin{tabular}{l}\hline
Cython generated C code that provides everything needed\\\hline
GMP C Library\\\hline
\end{tabular}
\end{center}
Another important problem with using SWIG is that
developers have to know C++ well and have the mental dexterity
to easily switch back and forth between C++ and Python.
Stein does not have this ability.

\subsubsection{Boost}
Like SWIG, Boost takes C++ code and generates C/C++ extension code and Python
wrappers around that extension code, which one can then use from the
Python interpreter. \SAGE does not use Boost because it is large and
difficult to redistribute and it is meant much more for wrapping
existing C++ libraries. Boost also uses a very complicated and
specialized build system.

\subsubsection{Psycho}
Psycho is a Python library that tries to guess the types of variables
in a running Python program, then generate compiled extension code
{\em on the fly} to optimize the running program. \SAGE does not use
Psycho, because it only runs on Intel x86 machines,
%[[what about
%OS X macs, windows?]]
it has large and
unpredictable memory requirements, and so much behindthescenes
guessing at this level for a serious research mathematics tool is
inappropriate.

%[[Nonetheless: quick tutorial and examples]]

Moreover, in my own experience with \sage, the
speedups from Psycho were not significant, except on code that had an
obvious C equivalent. Such code can be easily rewritten in Cython,
which always results in something that is faster, more robust, and
more reliable than what Pyscho produces, and which can be debugged
using gdb.

Though Psycho is not included with or used by \SAGE,
it is possible to try out Psycho by
installing it and putting the appropriate line at the top of a file.

%[[put a mathy example in with timings]]

\subsubsection{Ctypes}
Ctypes is included standard in Python 2.5, so it is clearly useful to
many Python users. It allows one to call library
functions defined in shared object libraries directory from
interpreted Python code. For the core of \SAGE
we have not found a use for this yet.
First, when implementing a basic type that needs to
be very fast, e.g., GMP integers, the entire implementation absolutely
must be compiled  writing it as a combination of Python and C
library calls will be way too slow. Second, it is often very painful
to convert Python data structures to C data structures and conversely,
and there are many opportunities for memory leaks and core dumps.

%[[example. like the ``install'' command in PARI.]]

\section{Examples}
+Python is an interpreted language and has no declared data types for
+variables. These features make it easy to write and debug, but Python
+code can sometimes be slow. Cython code can look a lot like Python,
+but it gets translated into C code (often very efficient C code) and
+then compiled. Thus it offers a language which is familiar to Python
+developers, but with the potential for much greater speed.
There are several ways to create and build Cython code in \SAGE.
\begin{enumerate}
@@ 1600,898 +1207,123 @@
for you to use in the notebook. Also, the output cell has
a link to the C program that was compiled to create
the .so file.
\item Below we will denote notebook sessions as follows:
\begin{verbatim}
{{{
 INPUT TEXT
///
 OUTPUT TEXT
}}}
...
\end{verbatim}
The triple braces delimit cells. You should {\em not}
actually type in the curly braces or triple forward
slashes.

\end{enumerate}
\item Create an .spyx file and attach or load it
from the command line.
This is similar to creating a {\tt \%cython} cell
in the notebook but works completely from the command
line.
+line (and not from the notebook).
\item Create a .pyx file in the \SAGE library, add it to {\tt
 SAGE_ROOT/devel/sage/setup.py} (note  you have to follow the
 template of how something else in there is done  we simply haven't
 had the time to properly document this process yet!), and type {\tt
 sage br} to build it. This is how much of the core arithmetic in
 the \SAGE library is implemented.


+\item Create a .pyx file in the \SAGE library and add a listing for it
+to the variable \code{ext_modules} in the file
+\code{SAGE_ROOT/devel/sage/setup.py}. For example, the file
+\code{SAGE_ROOT/devel/sage/sage/graphs/chrompoly.pyx} has lines
+%skip
+\begin{verbatim}
+ Extension('sage.graphs.chrompoly',
+ ['sage/graphs/chrompoly.pyx']
+ ), \
+\end{verbatim}
+in \code{setup.py}. Then type \code{sage ba} to build \sage with the
+new code.
\end{enumerate}
\subsection{Attaching or Loading {\tt .spyx} Files}
+
The easiest way to try out Cython without having to
learn anything about distutils, etc., is to create
a file with the extension {\tt spyx}, which stands
for ``SAGE Pyrex''. Try it now!
+for ``SAGE Pyrex'':
\subsection{Using the Command Line and a File}
\begin{enumerate}
\item Create a file {\tt hello.spyx}.
+\item Create a file \code{power2.spyx}.
\item Put the following in it:
\begin{verbatim}
def hello(name):
 """
 Print hello with the given name.
 """
 print("Hello %s"%name)
+def is2pow(n):
+ while n != 0 and n%2 == 0:
+ n = n >> 1
+ return n == 1
\end{verbatim}
\item Start the \SAGE commandline interpreter and load
the spyx file (this will fail if you do not have a C compiler
installed).
%skip
\begin{verbatim}
sage: load "hello.spyx"
Compiling hello.spyx...
sage: hello('World')
Hello World
+sage: load "power2.spyx"
+Compiling power2.spyx...
+sage: is2pow(12)
+False
\end{verbatim}
\end{enumerate}
Note that you can change {\tt hello.spyx}, then load
+Note that you can change \code{power2.spyx}, then load
it again and it will be recompiled on the fly.
You can also attach {\tt hello.spyx} so it is reloaded
whenever you make changes.
+You can also attach \code{power2.spyx} so it is reloaded
+whenever you make changes:
%skip
\begin{verbatim}
sage: attach "hello.spyx"
+sage: attach "power2.spyx"
\end{verbatim}

\subsection{An Optimization Example}
Imagine you are writing a program that frequently computes {\tt a*b + c}
for $a,b,c$ \SAGE integers. For example, you want to
do the following computation (timings on a Macbook Pro 2Ghz
under OS X):
+Cython is used for its speed. Here is a timed test on a 2.4 GHz iMac:
%skip
\begin{verbatim}
sage: a = 939393; b = 19393; c = 39390283
sage: time for _ in range(10000): a = a*b + c
CPU times: user 0.33 s, sys: 0.21 s, total: 0.53 s
+sage: time [n for n in range(10^5) if is2pow(n)]
+[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
+CPU time: 0.06 s, Wall time: 0.06 s
\end{verbatim}

There is no ``multiply then add'' operation built into \SAGE, and
you're very frustrated because you really want this to be as fast as
possible. There is possibly significant overhead in the computation
of {\tt a*b + c} in Python.

First create a new compiled ``multiply and add'' command in \SAGE.

%[[todo: change example below to use {\tt\_\_new\_\_}.]]
\begin{verbatim}
cimport sage.rings.integer
def muladd(sage.rings.integer.Integer a,
 sage.rings.integer.Integer b,
 sage.rings.integer.Integer c):
 """
 Compute a*b + c
 """
 cdef sage.rings.integer.Integer d
 d = sage.rings.integer.Integer()
 mpz_mul(d.value, a.value, b.value)
 mpz_add(d.value, d.value, c.value)
 return d
\end{verbatim}

Do this either by putting the above code in a spyx
file or pasting it into the notebook in a cell that
starts with {\tt\%cython}.

Now we can do the above computation more quickly:
+Now, the code in the file \code{power2.spyx} is valid Python, and if
+we copy this to a file \code{powerslow.py} and load that, we get the
+following:
%skip
\begin{verbatim}
sage: a = 939393; b = 19393; c = 39390283
sage: time for _ in range(10000): a = muladd(a,b,c)
CPU times: user 0.11 s, sys: 0.05 s, total: 0.15 s
+sage: load "powerslow.py"
+sage: time [n for n in range(10^5) if is2pow(n)]
+[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
+CPU time: 1.80 s, Wall time: 1.84 s
\end{verbatim}
+By the way, we could gain even a little more speed  cutting it down
+to 0.03 s  with the Cython version with a type declaration, by
+changing \code{def is2pow(n):} to \code{def is2pow(unsigned int n):}.
To squeeze out even more performance we can put the
entire loop in a single function.
\begin{verbatim}
cimport sage.rings.integer
def muladdloop(sage.rings.integer.Integer a,
 sage.rings.integer.Integer b,
 sage.rings.integer.Integer c,
 int n):
 cdef int i
 cdef mpz_t t
 cdef sage.rings.integer.Integer aa
 aa = sage.rings.integer.Integer(a)
 mpz_init(t)
 for i from 0 <= i < n:
 mpz_mul(t, aa.value, b.value)
 mpz_add(aa.value, t, c.value)
 mpz_clear(t)
 return aa
\end{verbatim}
+\section{Other Languages}
We then have
%skip
\begin{verbatim}
sage: a = 939393; b = 19393; c = 39390283
sage: time z=muladdloop(a,b,c,10000)
CPU times: user 0.11 s, sys: 0.00 s, total: 0.11 s
\end{verbatim}
This is a little faster. %[[no system overhead  what's going on?]]
+Since \sage is based on Python, it interfaces with C and C++, as well
+as other languages. See the Python documentation at
+\url{http://www.python.org/doc/} for more details; in particular,
+``Extending and Embedding the Python Interpreter'', available at
+\url{http://docs.python.org/ext/ext.html}, describes how to write C or
+C++ modules for use in Python.
\subsection{A simple loop example: Sum of squares}

The following \SAGE worksheet illustrates the above
sums of squares examples in the \SAGE Notebook.

Note, we use the notation:

%skip
\begin{verbatim}
{{{
 INPUT TEXT
///
 OUTPUT TEXT
}}}
\end{verbatim}
to denote cells in the notebook.

%[[todo put a check to avoid overflow below. maybe switch
%to using long long, or also use that...]]
\begin{verbatim}
{{{
two = int(2)
def sumsquarespy(n):
 return sum(i**two for i in xrange(1,n+1))
}}}
{{{
time v=[sumsquarespy(100) for _ in xrange(10000)]
///
#> CPU time: 0.49 s, Wall time: 0.49 s
}}}
{{{
%cython
# Next in cython.
def sumsquares(int n):
 cdef int i, j
 j = 0
 for i from 1 <= i <= n:
 j = j + i*i
 return j
}}}
{{{
time v=[sumsquares(100) for _ in xrange(10000)]
///
CPU time: 0.06 s, Wall time: 0.06 s
}}}
\end{verbatim}

\subsection{Example: 2 Power}
The following \SAGE notebook session illustrates how to implement
in Cython (entirely using the notebook) a function to quickly tell
if a C integer is a power of $2$. The resulting function is over 200
times faster in Cython than in Python, though the actual code
is {\em identical}.
\begin{verbatim}
{{{
%cython
def is2pow(unsigned int n):
 while n != 0 and n%2 == 0:
 n = n >> 1
 return n == 1
}}}
\end{verbatim}
\begin{verbatim}
{{{
time [n for n in range(10^5) if is2pow(n)]
///
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
CPU time: 0.05 s, Wall time: 0.06 s
}}}
\end{verbatim}
\begin{verbatim}
{{{
# The same program but in Python:
def is2pow(n):
 while n != 0 and n%2 == 0:
 n = n >> 1
 return n == 1
}}}
\end{verbatim}
\begin{verbatim}
{{{
time [n for n in range(10^5) if is2pow(n)]
///
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536]
CPU time: 13.04 s, Wall time: 15.08 s
}}}
\end{verbatim}
\begin{verbatim}
{{{
13.04/0.05
///
260.79999999999995
}}}
\end{verbatim}

%\subsection{Adding to \SAGE}

%\subsection{Your Own {\tt setup.py}}

\section{Cython Manual}

\subsection{Memory Usage}
Here is the memory usage of an integer mod int object on a 32bit system:
\begin{verbatim}
4 an int (the underlying value)
4 pointer to the parent
4 pointer to the modulus
4 pointer to the function call table
8 PyObject_HEAD (pointer to type object and reference count)
\end{verbatim}
So a single integer mod takes 24 bytes.

{\em IDEA: For a small field, one could store every element, then just have
lots of copies of pointers to the same thing, which would save a ton
of memory. }

\subsection{Forwarddeclaring extension types}
Extension types can be forward declared, which is necessary, e.g.,
if you have two extension types that need to refer to each other.
For example,
\begin{verbatim}
cdef class MyRing # forward declaration

cdef class MyElement:
 cdef MyRing parent

cdef class MyRing:
 cdef MyElement basis
\end{verbatim}

If you forward declare an extension type that has a base class, you
must specify the base class in both the forward declaration and its
subsequent definition. For example,

\begin{verbatim}
 cdef class A(B)

 ...

 cdef class A(B):
 # attributes and methods
\end{verbatim}

\subsection{Using cdef ModuleLevel Functions in Other Modules}
Suppose there is a cdef function in a Cython file \code{A.pyx} that is
not a method of a class, but you would like to use it from another
Cython module. There are (at least) two options.
One option is to make the functions methods of a cdef'd
class, then instantiate an object of that type in the other file
(this has no weird linking issues).
Another is to use ``public extern'' and link in the whole module, which we describe
below.

For example, the file \code{A.pyx} might contain
\begin{verbatim}
cdef int sqr_int(int x):
 return x*x
\end{verbatim}
You would like to call this Clevel method directly from a different Cython module.
To do so, {\em it is crucial} to declare the method public instead:
\begin{verbatim}
cdef public int sqr_int(int x):
 return x*x
\end{verbatim}
Then create a file \code{A.pxd} that contains
\begin{verbatim}
cdef extern int sqr_int(int x)
\end{verbatim}
Finally, to use \code{sqr_int} in another Cython module \code{B.pyx},
do the following:
\begin{verbatim}
cimport sage.rings.A # this is just an example absolute path
def myfunction():
 print sage.rings.A.sqr_int(5)
\end{verbatim}
Any time you access one Cython module from another one,
it's necessary that the compiler know to link in the other Cython
module (\code{A.pyx} in our example). Tell it to link
in \code{A.pyx} by including \code{A.pyx}
in the corresponding entry in \code{setup.py}. That
entry should look something like this:
\begin{verbatim}
 Extension('sage.modular.B',
 ['sage/modular/B.pyx', 'sage/rings/A.pyx'])
\end{verbatim}
\note{Linking as above means an entire copy of all the code in \code{A.pyx}
is compiled into the module for \code{B.pyx}, so \code{B.so} will be larger,
and the Clevel function you're accessing will {\em not} have access
to any modulescope variables in \code{A.pyx}. This can lead to weird
core dumps, depending on the nature of your function.}

Cython will generate a file \code{A.pxi} that also declares foo. You
can ignore that file.

{\bf NOTE:} Alternatively, you could put \code{include "path/to/A.pxi"}
in \code{B.pyx} and and directly use \code{sqr_int} in \code{B.pyx} with
no scoping:
\begin{verbatim}
def myfunction():
 print sqr_int(5)
\end{verbatim}
This is not recommended because it is confusing and
unpythonic to not have the scoping information.


\subsection{Making it possible to create weak references to extension classes}
By default, extension types do not support having weak references made
to them. You can enable weak referencing by declaring a C attribute of
type object called \code{__weakref__}. For example,
\begin{verbatim}
cdef class MyRing:
 """
 There is supposed to be at most one instance of this ring, so
 we keep a weak reference pointer to it, so it will be deleted
 from memory when there are no more user references to it.
 """
 cdef object __weakref__
\end{verbatim}

\subsection{Public and Private Variables}
To expose a variable to external Python, use \code{cdef public}, e.g.,
\begin{verbatim}
cdef class MyRing:
 cdef public Integer characteristic
\end{verbatim}
Use readonly if you want to enable only readonly access.
\begin{verbatim}
cdef class MyRing:
 cdef readonly Integer characteristic
\end{verbatim}

\subsection{Arbitrary Attributes}
Reminder: Like builtin types, Cython extension types don't have a
\code{__dict__} field, so you cannot assign arbitrary attributes to
them. That said, it is just a few lines to implement attribute
assignment, as illustrated by the following notebook session:
\begin{verbatim}
{{{
%cython
cdef class X:
 cdef object __dict__
 def __init__(self):
 self.__dict__ = {}
 def __setattr__(self, attr, val):
 self.__dict__[attr] = val
 def __getattr__(self, attr):
 try:
 return self.__dict__[attr]
 except KeyError:
 raise AttributeError, "object has no attribute '%s'"%attr
///
}}}
{{{
x = X()
///
}}}
{{{
x.a = 5
///
}}}
{{{
x.a
///
5
}}}
{{{
x.b
///
Traceback (most recent call last): x.b
 File "/Volumes/HOME/sagestable/local/lib/python2.5/", line 1, in

 File "/Volumes/HOME/.sage//spyx/sage20/sage20_0.pyx", line 14, in sage20_0.X.__getattr__
 raise AttributeError, "object has no attribute '%s'"%attr
AttributeError: object has no attribute 'b'
}}}
\end{verbatim}

\subsection{Special Methods}
Certain special methods {\em must} be defined in every extension class
in order to work. In particular, they do not derive from higher classes.
This can be very confusing if you are not aware of it. For
example, suppose you define two classes $A$ and $B$ as follows:
\begin{verbatim}
cdef class A:
 def __hash__(self):
 return 0

cdef class B(A):
 def __repr__(self):
 return "I am class B"
\end{verbatim}
Then $B$ will {\em not} inherit the \code{__hash__} method
from class $A$. If you want $B$ to be hashable, you {\em must}
define a hash method explicitly for $B$, e.g.,
\begin{verbatim}
cdef class B(A):
 def __hash__(self):
 return A.__hash__(B)
 def __repr__(self):
 return "I am class B"
\end{verbatim}

I do not know exactly which special methods do not get inherited.
Definitely \code{__hash__} and \code{__richcmp__} do not.



%\section{Gotchas}

%\subsection{How C extension classes different from Python classes}

%\subsection{When to use {\tt cdef class} versuses {\tt def class}
%in Cython}
%def allows multiple inheritance, but no clevel data structures.
%Maybe creation is faster. What exactly is the deal here?!

%\subsection{\SAGE's Pyrex is Patched}

%[[Record exactly how and why here. Post this to the
%mailing list every once in a while. ]]


%\subsection{There is no +=}
%[[solution make a C macro. need a sage.pxi that has all special
%convenience code like this in it, and is automatically included.]]

%\subsection{Notation for C for Loops}

%\subsection{Typos in Function Names and Variables}

\subsection{Doctests}
If \code{x.pyx} is a Cython file, then you can do
\begin{verbatim}
 sage t x.pyx
\end{verbatim}
to test all the examples in the documentation strings
in \code{x.pyx}.

\subsection{Signal Handling}
Suppose you have a Cython function like
\begin{verbatim}
def foo():
 cdef int n
 # Some long running code that gets compiled to pure C, e.g.,
 while True:
 n = n + 1

\end{verbatim}
If you call foo from the \SAGE interpreter, then when \SAGE
gets to the ``chunk of longrunning C code'' pressing controlC
will not interrupt the calculation. The only way to interrupt
the calculation is to completely kill your running copy of
\SAGE. What a wasteful painsurely there is a better way!

Indeed, if you simply rewrite the above code as follows, then
suddenly controlC will work fine!
\begin{verbatim}
def foo():
 cdef int n
 _sig_on
 while True:
 n = n + 1
 _sig_off
\end{verbatim}
\note{In a .pyx file which will be part of the \SAGE library,
you have to explicitly put something like
\code{include "../ext/interrupt.pxi"} in your file,
where you give in quotes the relative directory
to \code{SAGE_ROOT/devel/sage/sage/ext/interrupt.pxi}.}

In addition to SIGINT, any code wrapped in \code{_sig_on} and
\code{_sig_off} also traps SIGABRT, SIGALRM, SIGSEV, and SIGFPE.

{\bf IMPORTANT NOTES:}
\begin{enumerate}
 \item These {\em must} always come in pairs. E.g., if you have just
 a \code{_sig_off} without a corresponding \code{_sig_on}, then controlC
 later in the interpreter will segfault!

 \item Do {\em not} put these in the \code{__init__} method of
 a Cython extension class, or you'll get crashes.

 \item Do not put these around any Cython code that calls into
 the Python API (i.e., the generated C code should have no
 calls to the Python library)!!

 \item If you need to do a check of controlC inside a loop, e.g.,
 when constructing a matrix, put \code{_sig_check} instead of using
 \code{_sig_on} then \code{_sig_off}.
\end{enumerate}

\subsection{Introspection}
Users currently can't view the source code of functions
from the command line. This must change. Ideas for fixing
this problem.

If we can find the module name where foo is defined,
then check if the appropriate foo.pyx file is available.
Load it. Find the function. Figure out which if it
is defined twice. Show all if we can't figure it out.

If totally screwed and we can't even find the file, then
modify pyrex so it embeds extra info in the docstring.
Strip that out on displaying.

%\section{Automatic Coercion to and From C Data Types}
%Integer literals
%type checking
%Etc.
%How to do explicit casts

\subsection{Faster Iteration through a Python List or Tuple}
For example, define a function \code{mysum} in Cython as
follows. Note the trick of void
pointers to avoid Cython reference counting, etc:

\begin{verbatim}
def mysum(x):
 cdef object seq
 cdef int len,i
 cdef object item
 cdef object total
 total = 0
 cdef PyObject** X
 X = FAST_SEQ_UNSAFE(x)
 for i from 0 <=iX[i]
 total = total + item
 return (total)
\end{verbatim}
The call to \code{FAST_SEQ_UNSAFE} gets direct very very
unsafe access to the underlying array of PyObject's.

Somewhat surprisingly, this is actually {\em faster}
than Python's built in sum function:
%skip
\begin{verbatim}
sage: time sum(range(5*10^6))
CPU times: user 1.59 s, sys: 0.08 s, total: 1.67 s
12499997500000L
sage: time mysum(range(5*10^6))
CPU times: user 1.46 s, sys: 0.03 s, total: 1.49 s
12499997500000L
\end{verbatim}


In the next example, we illustrate the above idea by defining
several simplified versions of the \code{range} command.

\begin{verbatim}%skip
%cython

def f0(int n):
 return eval('[%s for _ in xrange(%s)]'%(n, n))

def f1(int n):
 v = []
 for i from 0 <= i < n:
 v.append(i)
 return v

def f2(int n):
 cdef int i
 v = [None] * n

 for i from 0 <= i < n:
 v[i] = i
 return v

def f3(int n):
 cdef int i
 w = [None] * n
 cdef void** X
 X = FAST_SEQ_UNSAFE(w)
 for i from 0 <= i < n:
 Py_DECREF(X[i])
 X[i] = PyInt_FromLong(i)
 return w

def f4(int n):
 cdef int i
 v = PyList_New(n)
 for i from 0 <= i < n:
 PyList_SET_ITEM(v, i,