Opened 12 years ago

Closed 10 years ago

Last modified 10 years ago

#7509 closed enhancement (fixed)

notebook -- make it possible to debug Python code in the notebook, e.g., something like pdb that works in the notebook

Reported by: was Owned by: boothby
Priority: major Milestone: sage-5.0
Component: interact Keywords:
Cc: novoselt, whuss Merged in: sage-5.0.beta3
Authors: William Stein, Wilfried Huss Reviewers: David Roe
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by was)

NOTE: "sage -coverage" claims less than 100% coverage, but there is 100% coverage. It's a bug in the coverage script that it doesn't count doctests marked with >>>.


Attachments (6)

trac_7509.patch (10.3 KB) - added by was 10 years ago.
ready to go
trac_7509.2.patch (10.5 KB) - added by was 10 years ago.
new version addressing the bug that was found ppurka -- apply only this
trac_7509.3.patch (10.5 KB) - added by was 10 years ago.
address bug introduced by bugfix. apply only this.
trac_7509_syntax_highlight.patch (4.5 KB) - added by whuss 10 years ago.
adds syntax highlighting
trac_7509.4.patch (10.9 KB) - added by was 10 years ago.
apply only this -- includes shrinking the stack; also prints a message if source code not known.
trac_7509.5.patch (11.2 KB) - added by was 10 years ago.
apply only this; addresses referee remarks.

Download all attachments as: .zip

Change History (36)

comment:1 Changed 12 years ago by was

  • Type changed from defect to enhancement

comment:2 Changed 12 years ago by was

Paste the following code into any notebook cell (in any version of sage ever) and you have a debugger.

class debug:
    """
    If you get a traceback in the notebook, type ``d=debug()`` right
    after the error to create a debugger object on that traceback.
    Using the debugger you can move and down the call stack, and 
    evaluate arbitrary code anywhere in the call stack (typically
    to inspect the value of variables).
    
        - printing ``d`` shows the call stack and some context
        - ``d.up(n)`` (or ``d.u(n)``) moves up `n` frames in 
          the call stack
        - ``d.down(n)`` (or ``d.d(n)``) moves down `n` frames
          in the call stack
        - ``d.list(n)`` (or ``d.l(n)``) displays `n` lines of source 
          code context around the current position in the stack trace
        - ``d("some code")`` executes the given code in the
          context of the current position in the stack trace
          
    Notes:
        - Input is not preparsed. 
        - You can define and work with many debug objects at the same time.
    """
    def __init__(self):
        import inspect, traceback
        self.stack = inspect.getinnerframes(sys.last_traceback)
        self.__curframe = len(self.stack) - 1
        self.tb = traceback.format_tb(sys.last_traceback)
        
    def curframe(self):
        return self.stack[self.__curframe][0]
        
    def frameno(self):
        return self.__curframe
        
    def __call__(self, line):
        locals = self.curframe().f_locals
        globals = self.curframe().f_globals
        try:
            code = compile(line + '\n', '<stdin>', 'single')
            exec code in globals, locals
        except:
            t, v = sys.exc_info()[:2]
            if type(t) == type(''):
                exc_type_name = t
            else: exc_type_name = t.__name__
            print '***', exc_type_name + ':', v
        
    def _highlight(self, line):
        from pygments import highlight
        from pygments.lexers import PythonLexer
        from pygments.formatters import HtmlFormatter
        return highlight(line, PythonLexer(), HtmlFormatter())        

    def __repr__(self):
        v = ['  ' + str(i) + ':  ' + ' '*(2*i) + 
                 ('\n'+' '*(2*i)).join(self.tb[i].splitlines()) 
                  for i in range(self.__curframe+1)]
        if len(v) < len(self.stack):
            v.append('  ............')
            v.append('  ' + str(len(self.stack)-1) + ':  (bottom frame)')
        s = '<b>Traceback:</b>' + self._highlight('\n'.join(v)) + self._list()
        return '<html>%s</html>'%s

    def up(self, n=1):
        self.__curframe -= n
        if self.__curframe < 0:
            self.__curframe = 0
    u = up
        
    def down(self, n=1):
        self.__curframe += n
        if self.__curframe >= len(self.stack):
            self.__curframe = len(self.stack)-1
    d = down
        
    def _list(self, n=3):
        curframe = self.curframe()
        filename = curframe.f_code.co_filename
        lineno = curframe.f_lineno
        import linecache
        code = ''.join('--> ' if i ==lineno else '    ' + 
                    linecache.getline(filename, i, curframe.f_globals) for
                    i in range(lineno-n, lineno+n+1))
        i = filename.rfind('site-packages/sage')
        if i != -1:
            fname = filename[i+len('site-packages/sage')+1:].rstrip('/')
            file = '<a href="/src/%s" target="_new">%s</a>'%(fname,fname)
        else:
            file = '<pre>%s</pre>'%filename
        t = """<b>Frame:</b> %s\n<b>File:  </b>%s<hr>%s<hr>"""%(
                 self.frameno(),                
                 file,self._highlight(code).strip())
        return t
                 
    def list(self, n=3):
        html(self._list(n))
    l = list

comment:3 Changed 12 years ago by was

Here's a proof concept interact, assuming one did:

d1 = debug()

at some point.

@interact
def _(command=[None, 'up','down'], code=''):
    if command == 'up':
        d1.up()
    if command == 'down':
        d1.down()
    if code:
        print "<html>>>> %s"%code
        d1(code)
        print '<hr></html>'
        
    print d1

comment:4 Changed 10 years ago by novoselt

  • Cc novoselt added
  • Report Upstream set to N/A

comment:5 Changed 10 years ago by was

  • Description modified (diff)

comment:6 Changed 10 years ago by was

Steps to try this out interactively in the notebook:

  1. install the patch and rebuild Sage.
  1. Do anything that causes a traceback in the notebook, e.g., EllipticCurve([0,0])
  1. Type debug() in any other notebook cell. You can move up and down the call stack, adjust the amount of context, and evaluate code in the box. These are the *only* features currently supported -- no setting of breakpoints, stepping through execution, etc. But this is a very useful first step.

Source code display for Cython code isn't implemented yet. That's for another ticket. I think it's OK to put off, since the command line debugger doesn't support it anyways.

comment:7 Changed 10 years ago by was

  • Status changed from new to needs_review

Changed 10 years ago by was

ready to go

comment:8 follow-up: Changed 10 years ago by ppurka

Should line 192 of your patch contain the string as r""" """?

        t = """%s<hr>> %s"""%(code, file)

I see that "< >" constructs are eaten away by the browser. An example code which you can use to debug your Debug() (pun intended :)) is the following:

ReedSolomonCode(15,2,GF(16,'a')).minimum_distance()

You need to go to stack frame 8, and look at line 396 of the code that is printed.

Otherwise, it functions very nicely!

comment:9 in reply to: ↑ 8 Changed 10 years ago by was

Replying to ppurka:

Should line 192 of your patch contain the string as r""" """?

        t = """%s<hr>> %s"""%(code, file)

No. r is *only* needed if your string contains backslashes and you don't want to escape them. It has nothing to do with HTML tags at all.

I see that "< >" constructs are eaten away by the browser.

Above t is an HTML string that I'm construting, and output within an <html> block for the notebook. It thus gets displayed properly.

An example code which you can use to debug your Debug() (pun intended :)) is the following:

ReedSolomonCode(15,2,GF(16,'a')).minimum_distance()

You need to go to stack frame 8, and look at line 396 of the code that is printed.

Ah, now I see what you mean. The stupid source code display needs to have some HTML escaping done, e.g., < turned to &lt;. There is a function in the notebook that does that properly, and I'll fix the patch accordingly. Thanks for finding a genuine bug. I'll also modify my example functions to include something like.

Otherwise, it functions very nicely!

Changed 10 years ago by was

new version addressing the bug that was found ppurka -- apply only this

comment:10 follow-up: Changed 10 years ago by was

ppurka -- please continue to review the patch now.

comment:11 in reply to: ↑ 10 Changed 10 years ago by ppurka

Replying to was:

ppurka -- please continue to review the patch now.

Yes. The listing is fixed now. But the hyperlink to the file is broken. I fail to see why this should happen, since the <hr> and filename are not escaped. (Tested this functionality on sagenb.org.)

Changed 10 years ago by was

address bug introduced by bugfix. apply only this.

comment:12 Changed 10 years ago by was

Doh -- fixed now.

Changed 10 years ago by whuss

adds syntax highlighting

comment:13 Changed 10 years ago by whuss

  • Cc whuss added

This is very nice und useful.

The patch [trac_7509_syntax_highlight.patch] adds syntax highlighting to the stack frame, and removes the first few frames, since they don't show anything interesting.

The <hr>> things are also fixed. Apply after the first patch [trac_7509.patch].

comment:14 follow-up: Changed 10 years ago by was

Hi whuss. Thanks. I'm reluctant to do syntax hilighting that way (the way you do in your patch, just on code), since it is very often completely wrong if we don't hilight the whole file. We need to hilight the whole file, then grab just the relevant part back. Interesting about removing the first few frames -- I was concerned about doing that; is this now consistent with %debug on the command line?

comment:15 Changed 10 years ago by roed

I think I agree that we should remove the first few frames. In the example I was playing with they were actually empty.

I don't know about syntax highlighting. I think it may be best to focus on in a later ticket.

Changed 10 years ago by was

apply only this -- includes shrinking the stack; also prints a message if source code not known.

comment:16 Changed 10 years ago by was

I made a new patch:

  • has the shrunk stack
  • prints something when source code not available (yet)
  • adds whuff as co-author.

comment:17 follow-up: Changed 10 years ago by roed

  • Reviewers set to David Roe
  • Status changed from needs_review to needs_work

This is while debugging the example

EllipticCurve([0,0])

If you type in a command (e.g. "print ainvs"), execute it, switch to a different frame and press enter again, nothing happens: it doesn't execute the print statement in the new frame. You can get it to execute by hitting the button though. If you type in a different command, then it starts executing upon pressing enter again.

There are doctest failures in listing, since you updated the functions.

Another small request would be to add a doctest to the evaluate function that demonstrates the case that a user types in a command to the text box which raises an error.

comment:18 in reply to: ↑ 17 Changed 10 years ago by was

Replying to roed:

This is while debugging the example

EllipticCurve([0,0])

If you type in a command (e.g. "print ainvs"), execute it, switch to a different frame and press enter again, nothing happens: it doesn't execute the print statement in the new frame. You can get it to execute by hitting the button though. If you type in a different command, then it starts executing upon pressing enter again.

This is a fundamental limitation of @interact right now. I can make a remark about this in the docstring.

I'll fix + improve the doctests and post a new patch.

There are doctest failures in listing, since you updated the functions.

Another small request would be to add a doctest to the evaluate function that demonstrates the case that a user types in a command to the text box which raises an error.

Changed 10 years ago by was

apply only this; addresses referee remarks.

comment:19 Changed 10 years ago by was

  • Status changed from needs_work to needs_review

comment:20 Changed 10 years ago by roed

Testing...

comment:21 Changed 10 years ago by roed

  • Status changed from needs_review to positive_review

All tests pass.

comment:22 in reply to: ↑ 14 Changed 10 years ago by whuss

Replying to was:

Hi whuss. Thanks. I'm reluctant to do syntax hilighting that way (the way you do in your patch, just on code), since it is very often completely wrong if we don't hilight the whole file. We need to hilight the whole file, then grab just the relevant part back.

I want to avoid highlighting the whole file, since this can be quite slow. On my computer it takes about 2 seconds to load and highlight graphs/generic_graph.py.

I think the main problem when one gets wrong highlighting is when the code fragment starts in the middle of a docstring. At ticket:12451 there is an improved patch which detects when the code fragments starts in the middle of a tripple quoted string, and does the correct highlighting in this case.

comment:23 Changed 10 years ago by jdemeyer

  • Milestone changed from sage-5.0 to sage-duplicate/invalid/wontfix
  • Resolution set to invalid
  • Status changed from positive_review to closed

Please report this patch to the SageNB developers

comment:25 Changed 10 years ago by jdemeyer

  • Component changed from notebook to interact
  • Milestone changed from sage-duplicate/invalid/wontfix to sage-5.0
  • Resolution invalid deleted
  • Status changed from closed to new

Apologies, this is actually independent of the notebook.

comment:26 Changed 10 years ago by jdemeyer

  • Status changed from new to needs_review

comment:27 Changed 10 years ago by jdemeyer

  • Authors set to William Stein, Wilfried Huss
  • Status changed from needs_review to positive_review

comment:28 Changed 10 years ago by jdemeyer

  • Merged in set to sage-5.0.beta3
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:29 follow-up: Changed 10 years ago by aranc

i think there is a minor typo in the docstring line 264

missing word "up": Using the debugger you can move <up> and down the stack frame and 

--

i'm not sure what is the preferred convention for fixing typos, i can make a patch for this isolated line.

comment:30 in reply to: ↑ 29 Changed 10 years ago by was

  • Description modified (diff)

Replying to aranc:

i think there is a minor typo in the docstring line 264

missing word "up": Using the debugger you can move <up> and down the stack frame and 

--

i'm not sure what is the preferred convention for fixing typos, i can make a patch for this isolated line.

See #12506. Please review!

Note: See TracTickets for help on using tickets.