Opened 12 years ago

Closed 12 years ago

Last modified 5 years ago

#7343 closed enhancement (fixed)

SageNB -- Add a Selenium test suite.

Reported by: timdumol Owned by: boothby
Priority: major Milestone: sage-4.2.1
Component: notebook Keywords:
Cc: was, mpatel Merged in:
Authors: Mike Hansen, Tim Dumol Reviewers: Mitesh Patel
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

Inclusion of a test suite will prevent regressions, and make development *much* easier.

Attachments (5)

trac_7343-selenium-tests.patch (127.9 KB) - added by timdumol 12 years ago.
New Selenium test suite.
trac_7343-selenium-tests.2.patch (103.7 KB) - added by timdumol 12 years ago.
The previous patch had some errors (*~ files, missing file).
trac_7343-selenium-tests.3.patch (104.7 KB) - added by timdumol 12 years ago.
Added a test to confirm that #7341 works.
trac_7343-selenium-tests.4.patch (106.8 KB) - added by timdumol 12 years ago.
Added "Edit worksheet" test. Made sure that no orphan processes are left behind by failed tests.
trac_7343-selenium-tests.5.patch (106.5 KB) - added by mpatel 12 years ago.
Tweaked a few tests. See the comment. Apply only this patch.

Download all attachments as: .zip

Change History (31)

Changed 12 years ago by timdumol

New Selenium test suite.

comment:1 Changed 12 years ago by timdumol

  • Authors set to Mike Hansen, Tim Joseph Dumol
  • Status changed from new to needs_review

Depends on #7309, #7310 and #7332. This is an initial version of the test suite. Many of these test cases were adapted and mildly modified from Mike Hansen's original Selenium test code.

More test cases to come, but I feel that having a test suite included asap is essential.

comment:2 Changed 12 years ago by timdumol

Running the test suite:

sage: from sagenb.testing.run_tests import run_tests; run_tests()

Note that this requires an instance of the Selenium server running on port 4444. The Selenium server can be downloaded from http://seleniumhq.org/download/ as part of the Selenium RC package and can be run with java -jar selenium-server.jar. This could not be included in the patch due to possible conflicts with the system binaries and libraries (Firefox, etc.).

comment:3 Changed 12 years ago by timdumol

A note: There is a failure in test_searching_worksheets. It is an actual bug -- the requisite javascript libraries are not loaded.

Changed 12 years ago by timdumol

The previous patch had some errors (*~ files, missing file).

Changed 12 years ago by timdumol

Added a test to confirm that #7341 works.

Changed 12 years ago by timdumol

Added "Edit worksheet" test. Made sure that no orphan processes are left behind by failed tests.

comment:4 Changed 12 years ago by mpatel

So far, this looks great! I'm still reading through the Selenium-RC documentation, but I think I can review this ticket formally tomorrow. At the risk of asking too soon: Can we specify the particular browser to test in run_tests()? Another potentially useful option, at least for Firefox: Use, e.g., firefox -P selenium-test -remote "openURL($*, new-tab)", to open multiple tabs instead of windows. I'll investigate.

Aside: It's nice that both Selenium and FunkLoad are build on Python's unittest framework. Of course, FunkLoad only simulates a browser, but I think we can attain approximate parity between the notebook's functional (i.e., with Selenium) and load (i.e., with FunkLoad) tests. Selenium's advantage is that it tests real-world browsers, but even with Selenium-Grid, we'd have difficulty (I think) simulating thousands of simultaneous notebook users.

comment:5 follow-up: Changed 12 years ago by mpatel

Possibly useful: HTMLTestRunner.py?

Changed 12 years ago by mpatel

Tweaked a few tests. See the comment. Apply only this patch.

comment:6 Changed 12 years ago by mpatel

  • Reviewers set to Mitesh Patel
  • Status changed from needs_review to positive_review

Version 5:

  • Fixes TestWorksheet.test_edit.
  • Changes TestWorksheet.test_7341, for now, to use
    sel.get_eval('window.evaluate_cell_introspection(1, null, null);')
    
    instead of
    self.tab('cell_input_1')
    
    For some reason, self.selenium.key_press_native(9) doesn't consistently send a TAB character in Firefox 3.5.3, at least for me. I think the problem is Selenium's.
  • Simplifies TestWorksheetsList.test_searching_for_worksheets. Is searching published worksheets broken in sagenb's tip? I see, e.g.,
            exceptions.IOError: [Errno 2] No such file or directory: '/home/.sage/sage_notebook.sagenb/home/pub/12/worksheet.html
    
    if I publish a worksheet, visit "Published," and try a search. Restarting the server fixes the problem.

Anyway, this is great work! To the extent it counts, my review is positive.

comment:7 Changed 12 years ago by mpatel

A sample Selenium-RC startup script:

#!/bin/bash
JAVA="/usr/java/latest/bin/java"
SEL_DIR="/home/apps/selenium/selenium-remote-control-1.0.1/selenium-serve
r-1.0.1"

FF_BROW="*firefox3"
FF_EXEC="/usr/lib64/firefox-3.5.3/firefox"
FF_PROF="/home/sage/selenium/firefox/selenium1"
OP_BROW="*opera"
OP_EXEC="/usr/lib/opera/opera"
CR_BROW="*chrome"
CR_EXEC="/usr/lib64/chromium-browser/chromium-browser"

BROW="$FF_BROW"
BROW_EXEC="$FF_EXEC"

OPTS=""
#OPTS="$OPTS -firefoxProfileTemplate $FF_PROF"
OPTS="$OPTS -singleWindow"
#OPTS="$OPTS -browserSessionReuse"
#OPTS="$OPTS -ensureCleanSession"
#OPTS="$OPTS -debug"
#OPTS="$OPTS -browserSideLog"
#OPTS="$OPTS -log test.log"
#OPTS="$OPTS -trustAllSSLCertificates"
#OPTS="$OPTS -forcedBrowserMode $BROW"
OPTS="$OPTS -forcedBrowserModeRestOfLine $BROW $BROW_EXEC"

$JAVA -jar "$SEL_DIR/selenium-server.jar" $* $OPTS 

Note: Opera 10 and Chromium are not supported.

comment:8 in reply to: ↑ 5 Changed 12 years ago by mpatel

Replying to mpatel:

Possibly useful: HTMLTestRunner.py?

See #7390.

comment:9 Changed 12 years ago by was

Unfortunately, I can't use any of this under OS X, because the sqlite libraries included in OS X and Firefox are incompatible:

After nearly two hours of trying to get this to work, I'm giving up for now. I guess Selenium is just broken on OS X for now. Hopefully the Firefox and/or Selenium devs will fix this asap. In the meantime, this is a vote from me to check out FunkLoad?: "Of course, FunkLoad? only simulates a browser, but I think we can attain approximate parity between the notebook's functional (i.e., with Selenium) and load (i.e., with FunkLoad?) tests." If nothing else, it would be really good to have a test suite that we can include and that anybody can trivially use with no confusing (or in my case, impossible) setup and configuration issues.

comment:10 Changed 12 years ago by was

Further comments about this:

(1) I tried to use FunkLoad? to test Sage (again in OS X) for a while, and got nowhere. It seems to be a mess and not very polished. Evidently it exists because some company wrote it and thought it might be useful to the community, so they released it. I'm dubious about FunkLoad?.

(2) No matter what, I think we need functional testing of the sage notebook that works on our build farms. There's no way Selenium can do that. I think we need something else to complement Selenium, possibly just something straightforward written in python using urllib, urllib2, etc.

comment:11 Changed 12 years ago by mpatel

Some browser drivers and/or simulators:

I apologize for any misclassifications. Some of the simulators can simulate JS, to a degree. I think these are all based on HtmlUnit.

It would be great if we could write tests in just one actively developed and supported framework/API but be able to run both truly functional and simulated tests. Selenium/WebDriver? may be(come) that framework.

comment:12 Changed 12 years ago by mpatel

By the way, this ticket depends on #7309, #7310, and #7332.

comment:13 Changed 12 years ago by mpatel

The Grinder, a Java load testing framework, has Jython bindings. It might be worth a closer look.

comment:14 Changed 12 years ago by mpatel

Notes:

  • I tried testing WebDriver's Python bindings to no avail. It may be best to wait for a post-merger release of Selenium.
  • Pylot test cases must be written in XML, which is not as expressive as Python.
  • Can we use Selenium Grid with a Linux virtual machine to work around the OS X problem, for now?
  • zope.testbrowser seems promising for light-weight functional testing.
  • For load tests, I think The Grinder and (yes) FunkLoad are worth a closer look. To get some data, I'll try to extend the simple FL test I ran recently to more complicated scenarios. The Grinder may well be a better choice, although it's written in Java, since it seems to be mature, well-documented, and actively developed. Moreover, we can write tests in Python, more or less.

comment:15 follow-up: Changed 12 years ago by mpatel

Good news, I think: We can use FunkLoad to create a new worksheet, enter/evaluate a cell, and check for updates every 0.25 seconds. We just need to send a [conditional] sequence of HTTP GET and POST requests. Unless there are objections, I'll try to carry this further, along the lines of the Selenium test suite. To the extent that FL is pure Python, its "functional" and load tests should run on multiple platforms.

Note: I did need to make one change to FL's redirect handling code:

  • src/funkload/FunkLoadTestCase.py

    old new class FunkLoadTestCase(unittest.TestCase 
    277277            thread_sleep()              # give a chance to other threads
    278278            while response.code in (301, 302, 303, 307) and max_redirect_count:
    279279                # Figure the location - which may be relative
     280                cookie = response.headers.get('Set-Cookie')
     281                if cookie:
     282                    self.setHeader('cookie', cookie)
    280283                newurl = response.headers['Location']
    281284                url = urljoin(url_in, newurl)
    282285                # save the current url as the base for future redirects

comment:16 Changed 12 years ago by mpatel

By the way FL, uses gnuplot to make graphs for bench reports. I suppose we could use matplotlib or Sage, instead, but I haven't examined the plotting code.

comment:17 in reply to: ↑ 15 Changed 12 years ago by timdumol

Replying to mpatel:

Good news, I think: We can use FunkLoad to create a new worksheet, enter/evaluate a cell, and check for updates every 0.25 seconds. We just need to send a [conditional] sequence of HTTP GET and POST requests. Unless there are objections, I'll try to carry this further, along the lines of the Selenium test suite. To the extent that FL is pure Python, its "functional" and load tests should run on multiple platforms. [snip]

Noting that FunkLoad? apparently requires patching before it is fully usable, what are the disadvantages to using zope.testbrowser, which seems to be better maintained and planned? I haven't tried it personally yet though, so I my impression may be wrong.

comment:18 Changed 12 years ago by mpatel

I also haven't used zope.testbrowser, but after reading through its documentation, I think it's better for "functional" testing than for load testing. In particular, it [very likely] has better cookie and form-handling capabilities than FL (actually, webunit). On the other hand, FL already has an infrastructure for concurrent testing and it can load/cache CSS/images/etc.

Of course, neither is a replacement for Selenium's truly functional tests. For "functional" tests alone, z.t may be a good choice. For load tests, FL seems to be a good Pythonic choice that can also go "functional." However, if we don't mind using other languages, Watir/Celerity? or The Grinder may be better. How important are the load tests? Perhaps we can build a load test framework on z.t?

comment:19 Changed 12 years ago by timdumol

I would think that load tests are quite important, since http://sagenb.org serves in excess of 10k users. I personally don't mind using other languages.

comment:20 follow-up: Changed 12 years ago by was

By the way, this ticket depends on #7309, #7310, and #7332.

Thanks for pointing this out. The sagenb-0.4.1 release I made that was going to to go into sage-4.2.1 contains #7343 but not #7309, #7310, and #7332. I'll merge those.

comment:21 Changed 12 years ago by mpatel

"If nothing else, it would be really good to have a test suite that we can include and that anybody can trivially use with no confusing (or in my case, impossible) setup and configuration issues."

To achieve this --- moreover, a fast, self-contained suite --- Python is a logical choice. But setup/config may be significantly simpler with simulated browsers. I suppose we need to cover three areas with up to three frameworks:

  1. In-browser - Se, Wa.
  2. Sim-browser - FL, Se 2.0, Wa, Zt.
  3. Load - FL, Gr.

comment:22 in reply to: ↑ 20 Changed 12 years ago by mpatel

Replying to was:

By the way, this ticket depends on #7309, #7310, and #7332.

Thanks for pointing this out. The sagenb-0.4.1 release I made that was going to to go into sage-4.2.1 contains #7343 but not #7309, #7310, and #7332. I'll merge those.

For 4.2.1, it may also be worthwhile to include one or more of these bug fixes: #7316, #7318, #7339, #7354, #7385.

comment:23 Changed 12 years ago by was

  • Milestone changed from sage-4.3 to sage-4.2.1
  • Resolution set to fixed
  • Status changed from positive_review to closed

I've merged this into sagenb-0.4.2 (sage-4.2.1)

comment:24 Changed 12 years ago by mpatel

A passing thought: We could reuse some functions in a "functional" Python test suite for a Sage Remote Access API (see, e.g., Google Documents List Data API). This would permit authenticated, programmatic access for manipulating worksheets. This may be an independent reason to use Zt.

comment:25 Changed 12 years ago by mpatel

On organizing tests:

Suppose we put all framework-dependent code (or as much as possible) in sagenb.testing.notebook_test_case, where we define NotebookTestCaseSelenium, NotebookTestCaseZopeTestbrowser, etc., as subclasses of NotebookTestCaseAbstract. Then, we can write tests under sagenb.testing.tests with a [largely] framework-independent, higher-level API. We can select a framework with which to run the tests in sagenb.testing.run_tests.

Of course, we can add specialized methods for particular frameworks. Moreover, over time, we can add or remove framework classes without discarding too many tests.

Thoughts?

comment:26 Changed 5 years ago by chapoton

  • Authors changed from Mike Hansen, Tim Joseph Dumol to Mike Hansen, Tim Dumol
  • Report Upstream set to N/A
Note: See TracTickets for help on using tickets.