Opened 10 years ago

Closed 4 years ago

#7298 closed enhancement (fixed)

use html5 video tag for animations

Reported by: whuss Owned by: whuss
Priority: major Milestone: sage-6.4
Component: graphics Keywords: animation, video
Cc: mhampton, niles, novoselt Merged in:
Authors: Martin von Gagern Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: ea5dfe2 (Commits) Commit: ea5dfe2f6b0ac9468962540e9afb7be33f869233
Dependencies: Stopgaps:

Description (last modified by gagern)

This ticket is about adding support for creating Ogg Theora or WebM videos from animation objects. The resulting video files are embedded into the notebook using the HTML 5 video tag.

The show method of animations now works more like the one of Graphics objects. Animations can now be embeddend in html fragments, for example using html.table(). By default animations are still created as animated GIFs. To get a modern video from an animation "a" use one of these:

a.show(mimetype="video/ogg")
a.show(format="webm")

The original ticket was designed to use libtheora and libogg from #7297 and only supported Theora. The currently attached branch builds on FFmpeg and supports a variety of formats, including Theora and WebM. Support for calling FFmpeg is already in place, so the focus here is on user interface, mainly integration with the show method and using HTML 5 video tags.

Attachments (3)

ogv.patch (8.4 KB) - added by whuss 10 years ago.
trac-7298-ogv.patch (8.8 KB) - added by whuss 9 years ago.
only apply this patch
libtheora_Mac-10.6.log (59.1 KB) - added by niles 9 years ago.
log file of failing install in Mac 10.6

Download all attachments as: .zip

Change History (81)

Changed 10 years ago by whuss

comment:1 Changed 10 years ago by whuss

  • Status changed from new to needs_review
  • Type changed from defect to enhancement

comment:2 Changed 10 years ago by jason

  • Cc mhampton added
  • Owner changed from whuss to jason
  • Report Upstream set to N/A

comment:3 Changed 10 years ago by jason

  • Owner changed from jason to whuss

comment:4 Changed 9 years ago by pang

  • Reviewers set to pang
  • Status changed from needs_review to positive_review

The "iterations" optional argument does not work, but I don't think it's needed, as html5 has video controls. I'd suggest to either remove this argument, or change the documentation of 'ogv' from:

  • iterations - integer (default: 0); number of

iterations of animation. If 0, loop forever.

to:

  • iterations - this argument is ignored, but kept for

compatibility with the 'gif' function

Otherwise, everything's fine. The issue above is absolutely minor, so I'm giving a positive review.

comment:5 Changed 9 years ago by mpatel

  • Authors set to Wilfried Huss
  • Merged in set to sage-4.5.2.alpha0
  • Resolution set to fixed
  • Reviewers changed from pang to Pablo Angulo
  • Status changed from positive_review to closed

I've updated the Author(s) and Reviewer(s) fields with full names. Please correct them, if I'm wrong.

comment:6 Changed 9 years ago by mpatel

By the way,

  • Mercurial patches for Sage should include the ticket number in the first line of the commit string.
  • The patch filename should include the ticket number. A common form is trac_7298-add_ogv_format.patch.

comment:7 follow-up: Changed 9 years ago by mpatel

  • Merged in sage-4.5.2.alpha0 deleted
  • Status changed from closed to needs_work

I'm getting a docbuild warning from sage/plot/animate.py:

:0: (WARNING/2) Literal block expected; none found.

I think the problem is that the ogv method's docstring ends with

        .. note::
           If libtheora is not installed, you will get an error
           message like this::

Could someone please fix this? For now, I'm removing ogv.patch from 4.5.2.alpha0.

Changed 9 years ago by whuss

only apply this patch

comment:8 in reply to: ↑ 7 Changed 9 years ago by whuss

  • Status changed from needs_work to needs_review

Replying to mpatel:

I'm getting a docbuild warning from sage/plot/animate.py:

:0: (WARNING/2) Literal block expected; none found.

I think the problem is that the ogv method's docstring ends with

        .. note::
           If libtheora is not installed, you will get an error
           message like this::

Could someone please fix this? For now, I'm removing ogv.patch from 4.5.2.alpha0.

I added the error message to the documentation in the new patch trac-7298-ogv.patch.

comment:9 follow-up: Changed 9 years ago by niles

  • Cc niles added

Hi,

I'd really like to use this, but I'm having trouble from the start . . .

sage: a = animate([sin(x + float(k)) for k in srange(0,2*pi,0.7)], xmin=0, xmax=2*pi, figsize=[2,1])
sage: a.ogv()
Error: libtheora does not appear to be installed. Saving an
animation to a Ogg Theora file or displaying an animation requires the
libtheora spkg, so please install it and try again.

sage: install_package('libtheora')
Traceback (most recent call last):
ValueError: Package is already installed. Try install_package('libtheora',force=True)

I did install both libogg and libtheora using install_package in sage 4.5.2

comment:10 in reply to: ↑ 9 ; follow-up: Changed 9 years ago by whuss

Replying to niles:

Hi,

I'd really like to use this, but I'm having trouble from the start . . .

sage: a = animate([sin(x + float(k)) for k in srange(0,2*pi,0.7)], xmin=0, xmax=2*pi, figsize=[2,1])
sage: a.ogv()
Error: libtheora does not appear to be installed. Saving an
animation to a Ogg Theora file or displaying an animation requires the
libtheora spkg, so please install it and try again.

sage: install_package('libtheora')
Traceback (most recent call last):
ValueError: Package is already installed. Try install_package('libtheora',force=True)

I did install both libogg and libtheora using install_package in sage 4.5.2

If the two packages are installed correctly the 'png2theora' command line utility should be in $SAGE_ROOT/local/bin.

And

sage: !png2theora --help

should give some output.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 9 years ago by niles

Replying to whuss:

Thanks for the reply; it seems that these are installed though:

If the two packages are installed correctly the 'png2theora' command line utility should be in $SAGE_ROOT/local/bin.

indeed:

$ which png2theora
/Applications/sage/local/bin/png2theora

And

sage: !png2theora --help

should give some output.

sage: !png2theora --help
png2theora 1.1
Usage: png2theora [options] <input>

The input argument uses C printf format to represent a list of files,
  i.e. file-%06d.png to look for files file000001.png to file9999999.png 
...

Other things I could check?

comment:12 in reply to: ↑ 11 ; follow-up: Changed 9 years ago by whuss

Replying to niles:

Other things I could check?

I don't know what the problem is. The code just creates a bunch of png files with the frames of the animation and then calls 'png2theora' to convert it an ogv file.

The relevant code is at the lines 405-415 of sage/plot/animate.py. You can try to remove the try/except block to see the real error message of check_call.

comment:13 in reply to: ↑ 12 ; follow-up: Changed 9 years ago by niles

Replying to whuss:

Replying to niles:

Other things I could check?

I don't know what the problem is. The code just creates a bunch of png files with the frames of the animation and then calls 'png2theora' to convert it an ogv file.

The relevant code is at the lines 405-415 of sage/plot/animate.py. You can try to remove the try/except block to see the real error message of check_call.

Thanks; the problem seems to be that, for me, check_call needs to be called with a list of inputs -- giving the command as a single string fails for some reason (png2theora returns its usage message, so probably it is not receiving the arguments in the expected format):

sage: check_call(["png2theora", '-o "tmp2.ogv"', '%08d.png'])
9 frames, 202x106
Compressing....                                          
./00000000.png
./00000001.png
./00000002.png
./00000003.png
./00000004.png
./00000005.png
./00000006.png
./00000007.png
./00000008.png
   
done.

0

but

sage: check_call(["png2theora", '-o "tmp2.ogv" %08d.png'])
png2theora 1.1
Usage: png2theora [options] <input>
...

and

sage: check_call('png2theora -o "tmp3.ogv" %08d.png')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
...
OSError: [Errno 2] No such file or directory

The last command is a simplified version of what appears in sage/plot/animate.py . . . does it really work for other people here? And if so, why?

comment:14 Changed 9 years ago by novoselt

  • Cc novoselt added

comment:15 Changed 9 years ago by niles

Apply trac-7298-ogv.patch

(This comment is for Patch Buildbot)

comment:16 in reply to: ↑ 13 Changed 9 years ago by niles

  • Reviewers changed from Pablo Angulo to Pablo Angulo, Niles Johnson
  • Status changed from needs_review to needs_work
  • Work issues set to pass list rather than single string to ``check_call``

Replying to niles:

Replying to whuss:

Replying to niles:

Thanks; the problem seems to be that, for me, check_call needs to be called with a list of inputs -- giving the command as a single string fails for some reason (png2theora returns its usage message, so probably it is not receiving the arguments in the expected format):

 sage: check_call(["png2theora", '-o "tmp2.ogv"', '%08d.png'])
 9 frames, 202x106
 Compressing....                                          
 ./00000000.png
 ./00000001.png
 ./00000002.png
 ./00000003.png
 ./00000004.png
 ./00000005.png
 ./00000006.png
 ./00000007.png
 ./00000008.png
    
 done.
 
 0

but

 sage: check_call(["png2theora", '-o "tmp2.ogv" %08d.png'])
 png2theora 1.1
 Usage: png2theora [options] <input>
 ...

and

 sage: check_call('png2theora -o "tmp3.ogv" %08d.png')
 ---------------------------------------------------------------------------
 OSError                                   Traceback (most recent call last)
 ...
 OSError: [Errno 2] No such file or directory

The last command is a simplified version of what appears in sage/plot/animate.py . . . does it really work for other people here? And if so, why?

This seems to be the offending code:

405	        cmd = 'cd "%s"; sage-native-execute png2theora -o "%s" -f %s  %%08d.png 2> /dev/null'%(d, savefile, int(100/delay))
406	        from subprocess import check_call, CalledProcessError
407	        try: 
408	            check_call(cmd, shell=True) 

I'm switching this to "needs work", since I think cmd needs to be a list of strings rather than a single string (see above).

comment:17 follow-up: Changed 9 years ago by niles

I tried working on this again, but now I'm having trouble installing the libtheora spkg with sage 4.6. For what it's worth, installing libogg went fine. Is this a known issue?

The last lines of the failing install (before the timing and other end messages) are:

...
make[2]: Nothing to be done for `install-data-am'.
make[2]: Nothing to be done for `install-exec-am'.
/bin/sh ./mkinstalldirs /Applications/sage/local/lib/pkgconfig
 /usr/bin/install -c -m 644 theora.pc /Applications/sage/local/lib/pkgconfig/theora.pc
 /usr/bin/install -c -m 644 theoradec.pc /Applications/sage/local/lib/pkgconfig/theoradec.pc
 /usr/bin/install -c -m 644 theoraenc.pc /Applications/sage/local/lib/pkgconfig/theoraenc.pc
cp: examples/.libs/png2theora: No such file or directory

comment:18 in reply to: ↑ 17 ; follow-up: Changed 9 years ago by whuss

Replying to niles:

I tried working on this again, but now I'm having trouble installing the libtheora spkg with sage 4.6. For what it's worth, installing libogg went fine. Is this a known issue?

I just tried to install libtheora on sage-4.6.2 (Debian 64bit), and it worked without problems.

The last lines of the failing install (before the timing and other end messages) are:

...
make[2]: Nothing to be done for `install-data-am'.
make[2]: Nothing to be done for `install-exec-am'.
/bin/sh ./mkinstalldirs /Applications/sage/local/lib/pkgconfig
 /usr/bin/install -c -m 644 theora.pc /Applications/sage/local/lib/pkgconfig/theora.pc
 /usr/bin/install -c -m 644 theoradec.pc /Applications/sage/local/lib/pkgconfig/theoradec.pc
 /usr/bin/install -c -m 644 theoraenc.pc /Applications/sage/local/lib/pkgconfig/theoraenc.pc
cp: examples/.libs/png2theora: No such file or directory

png2theora is one of the example programs included in libtheora and it is used by this patch to generate the ogv videos. This line means that it failed to build. Besides libtheora itself its only dependency is libpng. Did you get the line

checking for PNG... yes

in the configure log? Which platform are you using?

comment:19 in reply to: ↑ 18 ; follow-up: Changed 9 years ago by niles

Replying to whuss:

 checking for PNG... yes

Ah, I got

checking for PNG... no

I'm using Mac OS 10.6.6. Since I made my last post above, I have downloaded libtheora directly from their website and compiled png2theora successfully outside of Sage. I just retried installing from within Sage, and it still cannot find libpng, and still fails to build png2theora . . .

So the fix probably has to do with telling Sage how to accurately find libpng; is that right?

comment:20 in reply to: ↑ 19 Changed 9 years ago by whuss

Replying to niles:

Replying to whuss:

 checking for PNG... yes

Ah, I got

checking for PNG... no

I'm using Mac OS 10.6.6. Since I made my last post above, I have downloaded libtheora directly from their website and compiled png2theora successfully outside of Sage. I just retried installing from within Sage, and it still cannot find libpng, and still fails to build png2theora . . .

So the fix probably has to do with telling Sage how to accurately find libpng; is that right?

Yes. I have never developed on Mac OS, so I don't really know how to fix it. This is the autoconf check, used by libtheora to test for libpng:

dnl check for libpng
HAVE_PNG=no
if test "x$HAVE_PKG_CONFIG" = "xyes"
then
  PKG_CHECK_MODULES(PNG, libpng, HAVE_PNG=yes, HAVE_PNG=no)
fi
AC_SUBST(PNG_CFLAGS)
AC_SUBST(PNG_LIBS)

It is very simple, and since the original source builds for you, this should also work on Mac OS.

Maybe one could hard code the variables PNG_CFLAGS and PNG_LIBS in spkg-install, but there should be a better solution.

comment:21 follow-up: Changed 9 years ago by jason

I believe that we ship libpng, so it's weird that it didn't find it.

I know in matplotlib, though, on OSX, we look for libpng12. It seems that a long time ago, we kept picking up the wrong libpng on OSX, so now we look for libpng12. I believe the problem has been fixed since then, but the "fix" for matplotlib is still in the spkg.

comment:22 in reply to: ↑ 21 ; follow-up: Changed 9 years ago by whuss

Replying to jason:

I believe that we ship libpng, so it's weird that it didn't find it.

I know in matplotlib, though, on OSX, we look for libpng12. It seems that a long time ago, we kept picking up the wrong libpng on OSX, so now we look for libpng12. I believe the problem has been fixed since then, but the "fix" for matplotlib is still in the spkg.

I created a new spkg for libtheora, which checks for libpng12.

http://www.math.tugraz.at/~huss/spkg/libtheora-1.1.1.p1.spkg

It still works for me. Could someone test it on OSX?

Changed 9 years ago by niles

log file of failing install in Mac 10.6

comment:23 in reply to: ↑ 22 Changed 9 years ago by niles

Replying to whuss:

It still works for me. Could someone test it on OSX?

I'm afraid it's still failing for me--PNG is still not found. Here is the complete log file.

comment:24 follow-up: Changed 9 years ago by jason

I don't know autoconf. What exactly does PKG_CHECK_MODULES(PNG, libpng, HAVE_PNG=yes, HAVE_PNG=no) do? I think it's looking in $SAGE_ROOT/local/lib/pkgconfig, and you should have a libpng12.pc and a libpng.pc file in there. How exactly does PKG_CHECK_MODULES determine if libpng is installed?

comment:25 in reply to: ↑ 24 Changed 9 years ago by niles

Replying to jason:

I don't know autoconf. What exactly does PKG_CHECK_MODULES(PNG, libpng, HAVE_PNG=yes, HAVE_PNG=no) do? I think it's looking in $SAGE_ROOT/local/lib/pkgconfig, and you should have a libpng12.pc and a libpng.pc file in there. How exactly does PKG_CHECK_MODULES determine if libpng is installed?

well, I don't know any of this :) But I do indeed have libpng12.pc and libpng.pc in $SAGE_ROOT/local/lib/pkgconfig

comment:26 Changed 9 years ago by jhpalmieri

I tried installing on my mac, and I'm having a different problem. First I did sage -i libogg, and this seems to have worked. When I run sage -i libtheora or when I download and install the new spkg listed by whuss, I get these errors in the log file:

checking for Vorbis... no
*** Could not run Vorbis test program, checking why...
*** The test program failed to compile or link. See the file config.log for the
*** exact error that occured. This usually means Vorbis was incorrectly installed
*** or that you have moved Vorbis since it was installed.

and

*** The sdl-config script installed by SDL could not be found
*** If SDL was installed in PREFIX, make sure PREFIX/bin is in
*** your path, or set the SDL_CONFIG environment variable to the
*** full path to sdl-config.
configure: WARNING: *** Unable to find SDL -- Not compiling example players ***

It doesn't even check whether png is installed.

comment:27 Changed 9 years ago by jhpalmieri

I also notice that in config.log it says

HAVE_PKG_CONFIG=''

Would that explain why it was skipping the png test?

comment:28 follow-ups: Changed 8 years ago by ddrake

I think this patch is largely superseded by #11170, which provides an ffmpeg option for creating animations; by installing ffmpeg, you can create videos in Theora format, which makes the spkgs and patch here unnecessary. Also, ffmpeg supports far more formats, which makes it a more flexible solution.

Perhaps the focus here should be on changing the notebook so that animations are automatically displayed, much like images and animated GIFs are now. I would love to be able to create Theora or WebM videos with the animate() command and have them automatically displayed.

comment:29 in reply to: ↑ 28 ; follow-up: Changed 8 years ago by niles

Replying to ddrake:

I think this patch is largely superseded by #11170, which provides an ffmpeg option for creating animations; by installing ffmpeg, you can create videos in Theora format, which makes the spkgs and patch here unnecessary. Also, ffmpeg supports far more formats, which makes it a more flexible solution.

There is perhaps also the subtlety of patent and license issues -- there seem to be more severe limitations on the ways ffmpeg can be distributed, while png2theora is licensed under a relatively free BSD-style license . . . thus png2theora could conceivably be included with sage at some point, while ffmpeg is harder to imagine.

In any case, the patch at #11170 is quite useful, and doesn't suffer from the Mac-specific problems mentioned above.

comment:30 in reply to: ↑ 29 ; follow-up: Changed 8 years ago by jhpalmieri

Replying to niles:

There is perhaps also the subtlety of patent and license issues -- there seem to be more severe limitations on the ways ffmpeg can be distributed

Can you clarify this? I've seen this claim elsewhere, but I'm not sure what it means. On the ffmpeg web page it says that it is distributed under LGPL or GPL, depending on what components are being used, and this wouldn't put any extra restrictions on us. I guess the very last question on this page might be an issue, but it's not very clear to me.

comment:31 in reply to: ↑ 30 Changed 8 years ago by niles

Replying to jhpalmieri:

Replying to niles:

Can you clarify this?

Unfortunately no, I don't really understand the issues involved here. I just wanted to note that there do seem to be legal issues.

But you are right that ffmpeg itself seems intended to be free. The question (I think) is whether it infringes unintentionally on some patents (notably mpeg), but it may be that the answer to this question has implications only for commercial products. That question (whether or not mpeg patent infringement by ffmpeg might have real consequences for sage) would have to be sorted out before ffmpeg could be included.

comment:32 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:33 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:34 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:35 in reply to: ↑ 28 Changed 5 years ago by gagern

Replying to ddrake:

Perhaps the focus here should be on changing the notebook so that animations are automatically displayed, much like images and animated GIFs are now. I would love to be able to create Theora or WebM videos with the animate() command and have them automatically displayed.

See #16533 for a branch which does add the video tag, backed by ffmpeg. It also restores animation embedding in general, which appears to be broken at the moment, and it adds support for APNG as well.

comment:36 Changed 5 years ago by gagern

  • Branch set to u/gagern/ticket/7298
  • Modified changed from 06/26/14 07:48:53 to 06/26/14 07:48:53

comment:37 follow-up: Changed 5 years ago by gagern

  • Authors changed from Wilfried Huss to Martin von Gagern
  • Commit set to e34c05e0fce953cee590de42d67f1cbf409566f8
  • Dependencies set to #16533
  • Reviewers Pablo Angulo, Niles Johnson deleted
  • Status changed from needs_work to needs_review
  • Work issues pass list rather than single string to ``check_call`` deleted

I hope I'm not stepping on anybodies toes by commandeering this ticket here to carry my branch. And I hope I'm doing it the right way.

That branch is my <video> tag implementation, building on code from #16533. Once that is accepted, the modification to add the video tag is pretty small, and mostly involves some mime type fiddling.

I kept the table embedding stuff in there, since Wilfried Huss had it in his patch, and it seemed like a good idea but probably not worth a ticket all by itself.


New commits:

ac83bdbProper hyperlink formatting in animate documentation.
031b97eBase Animation.show on Animation.save, and pass more arguments.
0325ee2Create HTML to embed generated GIF in notebook.
e74146bSupport HTML5 video tag.
f6b64f1Allow embedding animations into tables.
e34c05eAlways pop video tag attributes from kwargs.

comment:38 in reply to: ↑ 37 Changed 5 years ago by niles

Replying to gagern:

I hope I'm not stepping on anybodies toes by commandeering this ticket here to carry my branch. And I hope I'm doing it the right way.

Well, it's been 3 years since the last activity here, I'm just happy that someone is paying attention to it again :)

comment:39 Changed 5 years ago by gagern

  • Status changed from needs_review to needs_work

comment:40 Changed 5 years ago by git

  • Commit changed from e34c05e0fce953cee590de42d67f1cbf409566f8 to c16f2753cd55784aea5f27d93ab184f15edf61c7

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

58b9c46minimal fix to restore functionality
16cbd2dMerge branch 'develop' into ticket/16533-stopgap
b3e656bgraphics_filename: return a tmp_filename() if not in EMBEDDED_MODE
69b285dRestore possibly positional arguments.
e3c5542Merge branch ticket/15515 into ticket/16608 to create ticket/16645.
9f30e94Always use graphics_filename instead of tmp_filename.
bb7307fPrevent doctest from leaking file into current working directory.
8af6a08Base Animation.show on Animation.save, and pass more arguments.
3beb8b1Create HTML to embed generated GIF in notebook.
c16f275Support HTML5 video tag.

comment:41 Changed 5 years ago by gagern

  • Dependencies changed from #16533 to #16645
  • Status changed from needs_work to needs_review

OK, I rebased this on my recent work on various other tickets, culminating in #16645. These are the changes that need reviewing in this ticket here.

comment:42 Changed 5 years ago by git

  • Commit changed from c16f2753cd55784aea5f27d93ab184f15edf61c7 to f0bfc49621c2a47d49bbd7315bba1d351dbb7997

Branch pushed to git repo; I updated commit sha1. New commits:

f0bfc49Allow embedding animations into tables.

comment:43 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:44 Changed 5 years ago by git

  • Commit changed from f0bfc49621c2a47d49bbd7315bba1d351dbb7997 to 2fb3d6ae91d0357d784d06d4d773222f74945597

Branch pushed to git repo; I updated commit sha1. New commits:

2fb3d6aMark test saving GIF as optional.

comment:45 Changed 5 years ago by git

  • Commit changed from 2fb3d6ae91d0357d784d06d4d773222f74945597 to 2f6a6703ba31efa50c24b8eab742151ac5adbd25

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

dc0067dBase Animation.show on Animation.save, and pass more arguments.
4e9101fCreate HTML to embed generated GIF in notebook.
e4807ccSupport HTML5 video tag.
2f6a670Allow embedding animations into tables.

comment:46 Changed 5 years ago by gagern

  • Dependencies #16645 deleted
  • Modified changed from 12/16/14 22:32:59 to 12/16/14 22:32:59

comment:47 Changed 5 years ago by gagern

  • Description modified (diff)

Changing description to reflect what this is currently about.

comment:48 Changed 5 years ago by gagern

The recent changes from #17234 interfere heavily with my work here.

I'm working on a rebase just now (which is somewhat annoying, considering that this ticket is waiting for review for 9 months now, and this is the third time I have to rebase it). But in some points I'm not certain how to integrate my goals with what #17234 did. For video, there are different formats to choose from, no single format will satisfy everybody, so I needed some flexibility there. On the other hand, almost all the different formats are the result of passing some PNG frames through ffmpeg, providing a suitable file name for the output, and then writing a HTML tag to embed that file. So all I need to know about a video format is a file name extension which ffmpeg will recognize and a mime type which the browser will recognize. I can even obtain one of these from the other using mimetypes.guess_extension resp. mimetypes.guess_type.

Now #17234 introduced the rich output type catalog, using yet another way to name types, and as far as I can see at the moment without hookups to either file name extensions or mime types. But what seems worse to me is that this catalog is an explicit list of allowed types, with no obvious room for extension at runtime.

  • Should I restrict users to e.g. Theora and WebM as video formats?
  • Should I add some OutputAnyVideo class to catch all cases?
  • Should I dynamically create new classes, one for every user-requested format?
  • How about support for more than one format, should we introduce a type which generates a bunch of common video formats (e.g. again Theora and WebM) to plug them into a single <video> tag so the browser can pick what he likes best, at the cost of increased conversion time?

Judging from e.g. the handling of LaTeX output, it looks as if I could simply generate HTML and have that recognized by the notebooks. I hope that's the case. I'll have to look whether embedding animations into tables still makes sense, but I hope it does.

comment:49 Changed 5 years ago by vbraun

I saw that the 2d animations are currently not in shaped and opened #17783, I didn't find your ticket.

The display backend (SageNB, IPython notebook, ...) declares which types it can display, and Sage then returns one (or None) of the supported types from _rich_repr_. There is no need for mime types (they are a bad fit for this problem anyways, whats the mime type for a sequence-of-pngs animation?). That is the only sane way to support multiple display backends, separation of concerns.

comment:50 follow-up: Changed 5 years ago by vbraun

I think the whole premise of "showing animations in a certain format" is misguided. Users care either about

  • actually seeing the animation play, or
  • saving an animation in a particular format to be used elsewhere.

But "showing in a particular format" is just bad UX, you are designing around technical abilities instead of what a user might want. This could also be said about the viewer=... argument for 3-d plots, but there we really don't have a truly good option so we need to give the user a way to choose between the devil and the deep sea. But GIF/APNG/Webm animations presumably all look the same to a human, so there is no point to expose those details when showing an animation.

comment:51 in reply to: ↑ 50 Changed 5 years ago by gagern

Replying to vbraun:

GIF/APNG/Webm animations presumably all look the same to a human

I very strongly disagree. Animations involving gradients, e.g. from 3D scenes, tend to look horrible in 255 color GIF. Animations with few frames and lots of change in between (like e.g. the first example from the animate docs) look horrible as video, due to heavy compression artifacts.

APNG looks as good as things can get, since we create everything else from a bunch of PNG frames and this is a lossless method. But for lengthy animations it might simply be too big to be feasible to transfer this. After all, my APNG assembler makes no effort to compress things using clever frame disposal combinations. It simply concatenates files, more or less.

Do I need to bundle up a few examples to support these claims, or will you accept my word or your own observations for this?

Users care either about […] saving an animation in a particular format to be used elsewhere.

The easiest way to do this, in my book, is have the animation show up in my notebook, then right-click and save to file. That works for cloud sessions as well as for local sessions as well as for sessions using some notebook server in my LAN. And I can always look at the animation first, make sure it looks the way I want it to, then save it to a local file once I'm satisfied. So the distinction between “view” and “save” is not as sharp as you claim.

comment:52 follow-up: Changed 5 years ago by vbraun

The IPython notebook doesn't support GIF/Webm and the browser probably not APNG, so what is this rightclick-save going to do? Nothing. Maybe save a single frame from the canvas where some javascript is paying the animation. Does not work.

comment:53 follow-up: Changed 5 years ago by vbraun

Distinguishing between lossy / lossless compression for graphics is reasonable, and doesn't require the user to know what webm&apng is. This is similar to preferring vector or raster graphics for 2-d plots. In #17234 I introduced a preference system for rich output, e.g.

sage: %display graphics vector
sage: plot(sin)

will prefer vector graphics formats when displaying the plot. This should probably be extended to cover lossy vs. lossless.

comment:54 in reply to: ↑ 52 ; follow-up: Changed 5 years ago by gagern

Replying to vbraun:

The IPython notebook doesn't support GIF/Webm

IPython is not the only notebook. I'm happy to use SageNB in almost all of my daily work, and have been happily using my own video-enabled branch on several occasions as well. If IPython doesn't support these, that's just one more reason to stick with SageNB for now, and hope that IPython learns to play along eventually.

and the browser probably not APNG

My Firefox supports APNG just fine, thank you very much.

so what is this rightclick-save going to do? Nothing. […] Does not work.

I tell you this is working, since I am doing this, on a fairly regular basis. (I very much hope you won't claim that this is just me pressing the space bar for heat-induced Ctrl.)

comment:55 in reply to: ↑ 54 ; follow-up: Changed 5 years ago by vbraun

Replying to gagern:

I very much hope you won't claim that this is just me pressing the space bar for heat-induced Ctrl.

You are calling show() to save a file. Do I need to say more? ;)

What any notebook *should* do is automatically download the file after you call save(), or at least show a download link. The fact that the saved file only lingers around on the remote server is yet another usability WTF. But thats a different ticket.

comment:56 in reply to: ↑ 53 Changed 5 years ago by gagern

Replying to vbraun:

Distinguishing between lossy / lossless compression for graphics is reasonable, and doesn't require the user to know what webm&apng is.

Distinguishing between the color-reduction losses of GIF and the frequency-domain compression losses of WebM and Theora does, in my opinion, indeed require some knowledge of formats. Sure, I won't mind if a polyfill-supported APNG becomes the lossless and shiny default, so that only the tech-savvy people ever have to make such a distinction.

I'd say that on the one hand, noone should be forced to make any format decision for elementary work (small 2D animations with few colors), but those who have the knowledge to choose a format should also have the power to get things displayed in their format of choice. I don't want to force anybody to save stuff to a file and open it in a different application just because showing it in line would offer too many choices.

By the way, how are chances that every browser-based notebook will reliably detect whether the browser supports WebM, Theora, or perhaps even something else which FFmpeg already supports but Sage does not know about?

I'd go a step further: there are many ways the animation composition might be tweaked. For ffmpeg videos you might control a million settings, bit rate probably the most important one. For GIF via ImageMagick there are a number of steps one can perform to achieve smaller and better files, reduction to a global color table foremost among these. I'd like to one day offer all of this power to the user, so he can pass ffmpeg command line switches or enable ImageMagick optimizations, whether he's saving to disk or viewing in browser. If you don't believe in “right click to save”, then you can still consider the “view in browser” as a preview to what will be saved to disk.

With the changes I had here, passing **kwds from show through save all the way to the format-specific methods, where additional options could be introduced as needed. Now if you claim that the user shouldn't even have control over whether the animation goes through FFmpeg or through ImageMagick, that approach doesn't make that much sense any more. I want to keep my power to choose.

comment:57 in reply to: ↑ 55 ; follow-up: Changed 5 years ago by gagern

Replying to vbraun:

What any notebook *should* do is automatically download the file after you call save()

No, I might be post-processing things remotely, from within the Sage session.

or at least show a download link. The fact that the saved file only lingers around on the remote server is yet another usability WTF. But thats a different ticket.

Sounds good, what ticket number is it?

Can we please also have an option preview to the method save so it will show an inlined preview of the file you've just saved and can download via the link? In that case, I'll simply switch to writing save instead of show and will be just as happy as I am now, since it does give show me the file inline, give me full control over the format, and lets me save via right-click or that link. I don't care about the name of the method, I care about what it does.

In case you want to continue this discussion on IRC, I'll try to stay around in #sagemath for a bit.

comment:58 in reply to: ↑ 57 Changed 5 years ago by vbraun

Replying to gagern:

Can we please also have an option preview to the method save so it will show an inlined preview of the file you've just saved

So instead of calling show() to save you want to call save() to show the animation? ;-)

comment:59 Changed 5 years ago by vbraun

Guess I missed you... maybe some time this weekend?

I created #17942 for save()-ing

comment:60 Changed 5 years ago by chapoton

  • Status changed from needs_review to needs_work

does not apply (and confuses the patchbot, see sage-devel)

comment:61 Changed 4 years ago by git

  • Commit changed from 2f6a6703ba31efa50c24b8eab742151ac5adbd25 to 025c6c12fdaf4ac34636712b2ad5da5c315ec6e4

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

025c6c1Trac #7298: Enable HTML5 video in the Sage Notebook

comment:62 Changed 4 years ago by gagern

  • Status changed from needs_work to needs_review

Yet another rebase. Perhaps it can be merged for Sage 6.7?

Right now, I support videos if the user explicitely requests a given format. When no format is specified, animated GIF remains the only option. I know Volker doesn't like it if people have to specify such low level stuff like file formats. And I wouldn't mind some sane automatic choices based on requests of a higher level, e.g. requests for full-color output or whatever. But getting that to work is far from trivial, so I'd ask you to postpone any fancy automatic format detection to a follow-up ticket, so that we can get the basic support in without getting lost in the details of fancy automatisms.

Right now, I don't support APNG as a format for show yet. #16650 should take care of that once this here gets merged. There might be some controversy ahead that way, so again I consider it best to get this bit here in before expanding on it.

Right now, I only support Sage Notebook, since that is all I use when creating animations. While I could test small examples using command line or the IPython notebook, I'm not using either on a regular basis, so I might be missing important aspects of how they operate. I'd prefer to merge this support for what I know, and postpone support for other backends. I have some ideas there, but I'd be asking more input from users of said backends, which could possibly delay things somewhat.

So please try to review this change with the scope I chose, keeping in mind that this is but the first step in a direction aiming for more high-level requests, more consistent support for various formats, and more support from various backends other than the Sage notebook. Feel free to create follow-up tickets if you want to start a discussion for one of these subsequent steps. If this gets merged early in the 6.7 cycle, we might get some of the other things in as well.

comment:63 follow-up: Changed 4 years ago by vbraun

I don't mind it if the initial ticket only supports SageNB. But OutputVideoAny is just a slap in the face of any attempt at sanity. The whole point of the rich output model is to separate different output types. The rich output container must not contain anything but the data. No mime types, no file extensions. The type is unique to the output. By lumping it into a single OutputAllVideoThatSageNBunderstands you destroy any hope of ever supporting other notebooks without reverting your branch.

comment:64 in reply to: ↑ 63 Changed 4 years ago by gagern

Replying to vbraun:

But OutputVideoAny is just a slap in the face of any attempt at sanity. The whole point of the rich output model is to separate different output types.

I can understand some of your concerns. To me, a supported output type is a promise. For most simple output types the promise is “if you give me a file of said type, I'll display it to the user”. But with video, it's not as simple. It's not the backend which decides what video formats are supported, but instead it's the browser or the system setup. So the promise I associate with OutputVideoAny is “if you give me a file of said type, I'll present it to a user in such a way that it either plays or the user gets notified about what went wrong”. Which in a browser context means “I'll create a HTML5 <video> tag for it”.

Due to this nature, OutputVideoAny should never be used to auto-guess supported formats. It doesn't mean “this client supports any video format”, but instead “this client can present any video format to the user, but that presentation might be in the form or an error message”. So the machinery shouldn't use it to throw arbitrary formats at the back end, but as a means to throw explicitly requested formats at it.

I wouldn't mind having specific formats. For video, a full file format description consists of a container format, a video codec format, and possibly a bunch of other constraints as well. So we'd have OutputVideoWebMVP8 and OutputVideoWebMVP9 and OutputVideoOggTheora and OutputVideoAviH264 and OutputVideoFlashH264 and OutputVideoMp4H264 and OutputVideoQuicktimeH264 and probably a hundred other combinations. (We could parse the list of formats FFmpeg supports.) And in my opinion, each of these would indicate a promise to actually support the given type. So we'd have to somehow detect support. Or state that such a type being listed with the supported types does not imply it's actually supported. I have no idea on how to do auto-detection, and having a bunch of maybe supported formats feels no different to my OutputVideoAny.

So do you have any suggestions of how to handle this situation in a better way? I understand your criticism from an idealistic point of view, but on the pragmatic side, unless one can do active format support detection, all formats are essentially the same to the backend, and a single representation for all of them avoids useless overhead.

The rich output container must not contain anything but the data. No mime types, no file extensions. The type is unique to the output. By lumping it into a single OutputAllVideoThatSageNBunderstands you destroy any hope of ever supporting other notebooks without reverting your branch.

It's not “output all video that Sage NB understands”. It's more like “output all video which can possibly be handled by a <video> tag”, combined with “output all video which gets opened by launching up the user's default media player”. For the <video> tag you need the MIME type. For the media player, you need the correct file extension. But if you consider the <video> element as the deliverable of the output, then the mime type is very much part of the data needed to assemble that.

One more comment: if there is one thing I can't stand about computer programs, it's if these try to be more clever than the user. The user should be in charge, and if the user says “I want to view this animation in that format”, the code should not second-guess that decision, should not forbid the operation just because the named format isn't in a list of known formats, or support by the browser or media player can't be verified up front. Please leave the users in charge of their application, not the other way round!

comment:65 Changed 4 years ago by vbraun

Supported means that we have code for trying to display that particular output type. There is not necessarily a guarantee that it'll work, you might be missing an external dependency (e.g. java). But if its unsupported then it can't possibly work because there is no code to do something with it.

At the same time you don't have to make a different output type for each sub-type, its ok to lump together formats that are likely to be supported at the same time (e.g. because they are common in some type of container). For example, there is no OutputImagePdf for each pdf version. But formats with a different mime type, say, aren't going to always be supported by the same player. In that sense their taxonomy is similar to mime types (but not exactly the same, e.g. support for multi-file output, jmol, ...) Just use your judgement. Is every video player going to be able to play animated gifs? I don't think so -> different output types.

On top of my head, I would suggest the following:

  • OutputVideoAnimatedGif
  • OutputVideoAnimatedPNG
  • OutputVideoWebm
  • OutputVideoMp4
  • OutputVideoFlash
  • OutputVideoQuicktime

comment:66 Changed 4 years ago by gagern

I fully agree with the Animated PNG format, that should be a type on its own. The OutputImageGif alreads includes animated GIF, according to its docstring. Should this be changed? Should the browser-based backends declare support for WebM, Ogg and MPEG4, even though the browser might be lacking support for either? Is there any way how a backend could, at some later time, indicate that indeed it does support a given video format?

comment:67 Changed 4 years ago by vbraun

I would prefer to make animated gif its own output type since many stand-alone image viewers will not show the animation. I just hadn't thought through the case of animations when I wrote it.

Ideally, browser-based backends would have some browser detection and then only return the actually supported output types. Though right now this is not implemented. Lacking that, you should return everything that a modern browser would support.

comment:68 Changed 4 years ago by gagern

Is Internet Explorer 11 a modern browser, even though it lacks WebM support? If you are tempted to answer this “no” and instead opt for MPEG-4 AVC as the codec with best cross-browser support, then please keep in mind that use of that codec might make the resulting content patent-encumbered, so I would very much like to keep at least one free alternative around. So for pragmatic reasons I'd declare recent releases of Chrome and Firefox as the only modern browsers here, and thus pretend that all web browsers support all three formats mentioned above. Or in other words, I'll go with the fact that the formats might be supported, which is all my OutputVideoAny approach did in any case. I'll adapt the code along these lines soon.

comment:69 Changed 4 years ago by vbraun

Fine with me. I'm not saying that it should behave differently than your OutputVideoAny, I'm just asking you to make it possible for other backends to rely on the output types.

comment:70 Changed 4 years ago by git

  • Commit changed from 025c6c12fdaf4ac34636712b2ad5da5c315ec6e4 to 00e1b487938287b694fa190645c1323c5a5be278

Branch pushed to git repo; I updated commit sha1. New commits:

00e1b48Trac #7298: Implement output types for various video container formats

comment:71 follow-up: Changed 4 years ago by vbraun

Thanks, looks much better.

I still dislike that HTML 5 tag attributes are attached to the rich output types, that makes no sense outside of a browser. The settings in question:

  • autoplay: Pointless, you ran animation.show() so obviously you want to see the animation
  • controls: Who would ever want no video controls? Hard to imagine any use case for that on a scientific data visualization platform.
  • loop: Seems desirable

So IMHO we should only have a loop=True optional keyword argument on the video output types, and no attrs={} dictionary.

comment:72 in reply to: ↑ 71 Changed 4 years ago by gagern

Replying to vbraun:

Thanks, looks much better.

Glad you say so. I'm still a bit concerned by the fact that a format supported by backend and FFmpeg still won't be supported only because it's not in our list.

I'm slightly worried by the fact that we save the video to some temporary file, only to copy it to the current working directory in a subsequent step, where SageNB can pick it up. This feels kind of wasteful. As far as I see, the temporary file used for the rich output never gets deleted (if this is wrong, where is the code doing that), which means a long-running SageNB process will accumulate temporary files without association to notebook cells, and might exhaust some tempfs due to this. Should this be a separate ticket?

I still dislike that HTML 5 tag attributes are attached to the rich output types, that makes no sense outside of a browser.

Well, most of the attributes map to other players as well. To support that claim, take e.g. xine which has -Z to suppress autoplay, --hide-gui and --no-gui to disable controls, and --loop or perhaps --loop=repeat to loop. So while the names of the attributes were taken from HTML5, the semantics can be found elsewhere as well.

The main problem is that each viewer has a different set of command line arguments to represent these options, and as long as we open stuff through xdg-open (which I consider the most likely approach for BackendIPythonCommandline on Linux), we have little chance of actually supporting any of this outside the browser in the near future.

The settings in question:

  • autoplay: Pointless, you ran animation.show() so obviously you want to see the animation

Use case: Creating the animation takes five minutes, so you work in some other window while Sage gets its act together. When you switch back you want to be able to view the animation from the beginning using a single click. Or perhaps you have a loop creating several animations, and want to show them one after the other. Controlling when an animation starts may be particularly important in an environment where you have an audience, and some effect visible in the animation which you don't want to divulge prematurely.

  • controls: Who would ever want no video controls? Hard to imagine any use case for that on a scientific data visualization platform.

I guess you are right there. I guess the main reason to have that in HTML is when you have your own controls written in JavaScript. Not something we need to worry about.

  • loop: Seems desirable

Mapping loop to e.g. the IPython command line backend sounds like it's going to be really tricky, but I think we simply have to live with the fact that some backends won't support some of the features.

So IMHO we should only have a loop=True optional keyword argument on the video output types, and no attrs={} dictionary.

If I could convince you with the autoplay use cases, should I add two keyword arguments? I guess so.

comment:73 Changed 4 years ago by vbraun

Temporary files are always a kludge, which is why the OutputBuffer is designed around in-memory buffers. In general there is no guarantee that the compute kernel and the notebook server share a common file system. Like all temporary files, they are eventually deleted by the separate sage-cleaner process.

There is a technical solution to not playing stuff in hidden browser tabs: http://caniuse.com/#feat=pagevisibility. No need to force the user to do it by hand.

If you want to show multiple animations one after the other then you really want a single animation with the frames concatenated. Presumably you can add animation objects...

If you don't want the audience to see the animation then don't evaluate the cell that shows the animation. Thats about equally disruptive in a presentation as having to click on something to play the animation. If the UI has a special "slides" mode (like the IPython notebook) then it should handle <video> in a way that does not disrupt the flow, you should only ever need forward/backward button on the presentation remote control.

comment:74 Changed 4 years ago by git

  • Commit changed from 00e1b487938287b694fa190645c1323c5a5be278 to ea5dfe2f6b0ac9468962540e9afb7be33f869233

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a8d97b7Trac #7298: Enable HTML5 video in the Sage Notebook
f6e6412Trac #7298: Implement output types for various video container formats
ea5dfe2Trac #7298: Drop support for the autoplay and controls attributes for video

comment:75 Changed 4 years ago by gagern

Rebased, and incorporated latest review comments in that last commit. Is this ready for merge now?

Perhaps in the long run we should have something like a prepare method for animation objects, which creates the video file without showing it, and on which one can call show or save later on, with minimal cost. But I'll leave that idea for later.

comment:76 Changed 4 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

comment:77 Changed 4 years ago by gagern

Thanks for the review. Now I can build on this for #16650.

comment:78 Changed 4 years ago by vbraun

  • Branch changed from u/gagern/ticket/7298 to ea5dfe2f6b0ac9468962540e9afb7be33f869233
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.