Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#13889 closed enhancement (fixed)

Better automatic backtrace

Reported by: vbraun Owned by: jdemeyer
Priority: major Milestone: sage-5.7
Component: build Keywords:
Cc: SimonKing Merged in: sage-5.7.beta0
Authors: Volker Braun Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #13881, #13866 Stopgaps:

Description (last modified by vbraun)

When Sage encounters a SIGSEGV, it uses the glibc backtrace. This is sometimes seen in doctests that are not reproducible, making it hard to debug. The aim of this ticket is to return better automatic stack traces by automatically attaching gdb and running scripts in gdb's Python interpreter. This will will pinpoint the cause much easier, especially in a debug build.

The log is printed to stdout and automatically saved to a log file in $DOT_SAGE/crash_logs with logs older than a week automatically deleted.

Apply

Attachments (4)

sage_crash_Ho8Cji.log (9.5 KB) - added by jdemeyer 7 years ago.
Sample log
trac_13889_crime_scene_inspector.patch (9.2 KB) - added by vbraun 7 years ago.
Updated patch
trac_13889_enhanced_backtrace.patch (1.9 KB) - added by jdemeyer 7 years ago.
Updated patch
13889_disable_osx.patch (828 bytes) - added by vbraun 7 years ago.
Initial patch

Download all attachments as: .zip

Change History (54)

comment:1 Changed 7 years ago by vbraun

  • Description modified (diff)

comment:2 Changed 7 years ago by vbraun

  • Description modified (diff)

comment:3 Changed 7 years ago by vbraun

  • Dependencies set to #13881

comment:4 Changed 7 years ago by vbraun

  • Authors set to Volker Braun
  • Status changed from new to needs_review

comment:5 Changed 7 years ago by SimonKing

  • Cc SimonKing added

comment:6 follow-up: Changed 7 years ago by jdemeyer

I don't think I know enough about gdb to review this completely, but some remarks anyway:

  1. There is a double "and" in
    "Sage will now try to attach the debugger to get more information and\n"
    "and then terminate.\n"
    
  1. Why should the CSI script kill the Sage process? Couldn't you use a different IPC mechanism, such as a simple wait() (or a different signal or a pipe if you want)? The sentence "The target process is frozen while this script runs and resumes when it is finished." in sage-CSI is currently wrong.
  1. I preferred how the backtrace currently appears before the "Unhandled SIGSEGV: ..." message. When something goes wrong, that message stands out at the end of the console output. I think all those backtraces might confuse less experienced Sage users.
  1. In run_gdb(), stderr will always be None since you didn't redirect it. That's fine, so remove
    if stderr != None: 
        result.append('An error ocurred:') 
        result.append(stderr)
    
  1. The Popen command should probably be wrapped in a
    try:
        cmd = Popen(...)
    except OSError:
        return ""
    

block to handle the case that gdb isn't installed (and I guess empty backtraces should not be saved to a file).

Last edited 7 years ago by jdemeyer (previous) (diff)

comment:7 in reply to: ↑ 6 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

I don't think I know enough about gdb

At least you certainly know more than I do. So, sorry for my unqualified comments...

  1. I preferred how the backtrace currently appears before the "Unhandled SIGSEGV: ..." message. When something goes wrong, that message stands out at the end of the console output.

Yes, when an error in Python is raised, one first sees the backtrace and then the error message. If Sage crashes, it makes sense to show first the backtrace and then the error message as well.

I think all those backtraces might confuse less experienced Sage users.

Well, a less experienced Sage user will hopefully never see such backtraces.

What I mean is: Sage is supposed to run without crashes when doing low-level stuff. Hence, crashes are supposed to only occur if the user creates new buggy Cython programs in Sage, or tries to tweak Sage until it exposes its existing bugs. And I think a user who does such "real" programming and developing is experienced enough to not be confused by a backtrace.

comment:8 in reply to: ↑ 7 Changed 7 years ago by jdemeyer

Replying to SimonKing:

Well, a less experienced Sage user will hopefully never see such backtraces.

What I mean is: Sage is supposed to run without crashes when doing low-level stuff.

Unfornately, "hopefully" and "should" aren't always applicable. There are plenty of cases where seemingly innocent calculations produce the dreaded "Unhandled SIGSEGV" message.

comment:9 follow-up: Changed 7 years ago by vbraun

It makes sense to have the "Unhandled SIGSEGV..." message at the end. I don't think trying to get back from the debugger into the main program is a good idea. Sage is currently in a signal handler after something bad happened, there are no guarantees that anything is still working. In fact, just calling printf() from the signal handler is, strictly speaking, illegal. But the sage-CSI script can easily print the message.

The "The target process is frozen while this script runs and resumes when it is finished." is correct (unless you add --kill, what did you expect). In fact you can use the sage-CSI to see the status of a long-running computation without disturbing it if you don't pass --kill.

comment:10 in reply to: ↑ 9 Changed 7 years ago by jdemeyer

Replying to vbraun:

It makes sense to have the "Unhandled SIGSEGV..." message at the end. I don't think trying to get back from the debugger into the main program is a good idea. Sage is currently in a signal handler after something bad happened, there are no guarantees that anything is still working.

True, but what's your point? Your execlp() might also fail for this reason. While fork() and wait() are basically pure system calls, so much less probable to fail.

So I stand by my proposal to remove the sleep(10) kludge and replace it by a wait(). Then you don't need to kill the parent process.

comment:11 Changed 7 years ago by vbraun

I don't mind going the wait() route. Its still illegal to use printf() in the signal handler but seems to work more often than not.

comment:12 Changed 7 years ago by vbraun

I've addressed all issues. Also, I don't see why you need to be an expert in GDB to review this ticket.. ;-)

comment:13 Changed 7 years ago by jdemeyer

The GDB thing doesn't quite work for me, don't know why:

------------------------------------------------------------------------
Attaching to process id 8267.
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
(gdb) Hangup detected on fd 0
error detected on stdin
Saved trace to /home/jdemeyer/.sage/crash_logs/sage_crash_WM1Ga9.log

------------------------------------------------------------------------

comment:14 Changed 7 years ago by vbraun

Gdb 6.7.1 is ancient (vintage 2007). I doubt it has any Python supporty...

comment:15 Changed 7 years ago by jdemeyer

Fair enough, compiling gdb-7.5 now...

comment:16 Changed 7 years ago by jdemeyer

Please have a look at this logfile, the last part looks broken.

It was generated by

./sage -c "from sage.tests.interrupt import *; unguarded_abort()"

comment:17 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_review to needs_work

comment:18 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:19 Changed 7 years ago by vbraun

What happens if you import site into the python interpreter in gdb?

(gdb) python import site
(gdb) python import sys               
(gdb) python print sys.modules['site']
<module 'site' from '/home/vbraun/opt/sage-5.5.rc1/local/lib/python/site.pyc'>

Did you build gdb with python enabled? This should be the default but obviously requires Python to be installed in Gentoo.

(sage-sh) vbraun@volker-desktop:spkg$ ldd /usr/bin/gdb
	linux-vdso.so.1 =>  (0x00007fffd2584000)
	libreadline.so.6 => /home/vbraun/opt/sage-5.5.rc1/local/lib/libreadline.so.6 (0x00007faa4059e000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x000000325f400000)
	libncurses.so.5 => /lib64/libncurses.so.5 (0x000000327a400000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003275400000)
	libz.so.1 => /home/vbraun/opt/sage-5.5.rc1/local/lib/libz.so.1 (0x00007faa4035f000)
	libm.so.6 => /lib64/libm.so.6 (0x000000325e400000)
	libdl.so.2 => /lib64/libdl.so.2 (0x000000325dc00000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x000000325d800000)
	libutil.so.1 => /lib64/libutil.so.1 (0x0000003276800000)
	libpython2.7.so.1.0 => /home/vbraun/opt/sage-5.5.rc1/local/lib/libpython2.7.so.1.0 (0x00007faa3ff7f000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x0000003262800000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x0000003264000000)
	libc.so.6 => /lib64/libc.so.6 (0x000000325d400000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x000000325f000000)
	/lib64/ld-linux-x86-64.so.2 (0x000000325d000000)
	librt.so.1 => /lib64/librt.so.1 (0x000000325e000000)

comment:20 Changed 7 years ago by vbraun

On further thought, it seems that your OS Python was built without weakref, but Sage Python is of course built with weakref.

comment:21 Changed 7 years ago by jdemeyer

Even just starting gdb within a Sage shell gives me:

(sage-sh) jdemeyer@arcanis:sage-5.6.beta3$ gdb
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/os.py", line 398, in <module>
    import UserDict
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/UserDict.py", line 83, in <module>
    import _abcoll
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/abc.py", line 8, in <module>
    from _weakrefset import WeakSet
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/_weakrefset.py", line 5, in <module>
    from _weakref import ref
ImportError: No module named _weakref
GNU gdb (Gentoo 7.5 p1) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>.
(gdb)

Aren't there always going to be problems if gdb uses system Python instead of Sage python?

comment:22 Changed 7 years ago by vbraun

Well I only tried it on my machine, and there it works. Both Fedora 18 and Sage use Python 2.7.3, so it is no surprise that it works.

Its curious that you get the backtrace immediately, line 14 and 15 in your log were printed by Python (line 36 and 37 in sage-CSI). I guess interactive use tries to load the os module and we are screwed. Though we could at least use that to give a better error message.

We could unset LD_LIBRARY_PATH but you also need Cython to get the Cython source information. So I guess we should point users to the gdb spkg #13866.

comment:23 Changed 7 years ago by vbraun

I've added some checking for gdb errors to point users towards he Sage gdb spkg. Untested since it does work for me ;-)

comment:24 Changed 7 years ago by jdemeyer

  • Dependencies changed from #13881 to #13881, #13866

I do indeed get that message:

An error was sent to stderr:
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/os.py", line 398, in <module>
    import UserDict
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/UserDict.py", line 83, in <module>
    import _abcoll
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/abc.py", line 8, in <module>
    from _weakrefset import WeakSet
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/_weakrefset.py", line 5, in <module>
    from _weakref import ref
ImportError: No module named _weakref
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
warning: File "/home/jdemeyer/local/lib64/libstdc++.so.6.0.16-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/os.py", line 398, in <module>
    import UserDict
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/UserDict.py", line 83, in <module>
    import _abcoll
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/abc.py", line 8, in <module>
    from _weakrefset import WeakSet
  File "/usr/local/src/sage-5.6.beta3/local/lib/python/_weakrefset.py", line 5, in <module>
    from _weakref import ref
ImportError: No module named _weakref
Error while executing Python code.

The GDB/Python support seems to not work.
Install the gdb spkg (sage -f gdb) for enhanced tracebacks.

comment:25 Changed 7 years ago by jdemeyer

It would be nice if we could somehow force system gdb to use Sage's version of Python.

comment:26 Changed 7 years ago by jdemeyer

It seems that gdb doesn't pick up Sage's Python if the library versions don't match. Part of the strace from "gdb":

open("/usr/local/src/sage-5.6.beta3/local/lib/libpython2.6.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/src/sage-5.6.beta3/local/lib/R/lib/libpython2.6.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/jdemeyer/local/lib/libpython2.6.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/jdemeyer/local/lib32/libpython2.6.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/jdemeyer/local/lib64/libpython2.6.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libpython2.6.so.1.0", O_RDONLY|O_CLOEXEC) = 3

But imports come from Sage:

open("/usr/local/src/sage-5.6.beta3/local/lib/python/os.py", O_RDONLY) = 4

Changed 7 years ago by jdemeyer

Sample log

comment:27 Changed 7 years ago by jdemeyer

With the gdb spkg: sage_crash_Ho8Cji.log

comment:28 Changed 7 years ago by jdemeyer

Simply putting a symbolic link

ln -s libpython2.7.so.1.0 libpython2.6.so.1.0

in $SAGE_ROOT/local/lib made system-gdb work fine.

comment:29 Changed 7 years ago by jdemeyer

Which make me wonder: should we change the Python spkg to make symbolic links like to one above (say, based on which libraries exist in /usr/lib).

Last edited 7 years ago by jdemeyer (previous) (diff)

comment:30 Changed 7 years ago by jdemeyer

  • Owner changed from GeorgSWeber to jdemeyer

The script correctly detects if gdb is not installed:

------------------------------------------------------------------------
Attaching to process id 22967.
Unable to start gdb (not installed?)
The GDB/Python support seems to not work.
Install the gdb spkg (sage -f gdb) for enhanced tracebacks.
------------------------------------------------------------------------

comment:31 follow-up: Changed 7 years ago by vbraun

Replying to jdemeyer:

Which make me wonder: should we change the Python spkg to make symbolic links like to one above (say, based on which libraries exist in /usr/lib).

No, definitely not. Library versioning is there for a reason. In the case of GDB you might get away with it since GDB just runs an embedded Python, but anything that really touches Python internals might get very upset.

But maybe we can LD_PRELOAD libpython just for gdb, does that work?

comment:32 Changed 7 years ago by vbraun

The log with the gdb spkg (sage_crash_Ho8Cji.log) shows the intended output. There is an error at the end because you don't have debugging symbols for system libraries. We could suppress that but then it can also be a useful reminder if you suspect that the bad stuff happens in glibc, say.

comment:33 in reply to: ↑ 31 Changed 7 years ago by jdemeyer

Replying to vbraun:

Replying to jdemeyer:

Which make me wonder: should we change the Python spkg to make symbolic links like to one above (say, based on which libraries exist in /usr/lib).

No, definitely not. Library versioning is there for a reason. In the case of GDB you might get away with it since GDB just runs an embedded Python, but anything that really touches Python internals might get very upset.

But maybe we can LD_PRELOAD libpython just for gdb, does that work?

Yes, it does, at least for Python-2.6.

When system Python is 2.4, it doesn't work. I don't have a system with Python 2.5 and a sufficiently recent GDB.

comment:34 Changed 7 years ago by vbraun

I've added LD_PRELOAD / DYLD_INSERT_LIBRARIES and better diagnostic for gdb errors.

comment:35 Changed 7 years ago by vbraun

  • Status changed from needs_work to needs_review

comment:36 follow-up: Changed 7 years ago by jdemeyer

Looks good so far. I'm going to test it on a few different systems (in particular OS X).

Do you agree with 13889_sagelib_review.patch?

comment:37 in reply to: ↑ 36 Changed 7 years ago by vbraun

Replying to jdemeyer:

Do you agree with 13889_sagelib_review.patch?

Yes!

comment:38 Changed 7 years ago by jdemeyer

Could you add

libpython = os.path.join(env['SAGE_LOCAL'], 'lib', libpython)

since libpython is returned as relative path and causes gdb to fail to start on OS X.

But even with this change, the hack doesn't work on OS X.

System gdb:

(sage subshell) bsd:sage-4.8 jdemeyer$ DYLD_INSERT_LIBRARIES="$SAGE_LOCAL/lib/libpython2.7.dylib" /usr/bin/gdb
dyld: Symbol not found: _environ
  Referenced from: /Users/jdemeyer/sage-4.8/local/lib/libpython2.7.dylib
  Expected in: flat namespace
 in /Users/jdemeyer/sage-4.8/local/lib/libpython2.7.dylib
Trace/BPT trap

gdb spkg but compiled (on purpose) with the wrong Python version:

(sage subshell) bsd:sage-4.8 jdemeyer$ DYLD_INSERT_LIBRARIES="$SAGE_LOCAL/lib/libpython2.7.dylib" gdb
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jdemeyer/sage-4.8/local/lib/python/os.py", line 398, in <module>
    import UserDict
  File "/Users/jdemeyer/sage-4.8/local/lib/python/UserDict.py", line 83, in <module>
    import _abcoll
  File "/Users/jdemeyer/sage-4.8/local/lib/python/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
  File "/Users/jdemeyer/sage-4.8/local/lib/python/abc.py", line 8, in <module>
    from _weakrefset import WeakSet
  File "/Users/jdemeyer/sage-4.8/local/lib/python/_weakrefset.py", line 5, in <module>
    from _weakref import ref
ImportError: No module named _weakref
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin10.8.0".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)

Changed 7 years ago by vbraun

Updated patch

comment:39 Changed 7 years ago by vbraun

I've added the full path name since it can't hurt.

Upon reflection, I think this is the reason why it does not and cannot work on OSX. In order to protect their precious DRM, Apple has disabled much of the ptrace API. A consequence is that /usr/bin/gdb needs special permissions (it is setgid on OSX, but a normal program on Linux) to fully function. And the dynamic linker disallows library interposition for setgid binaries. In fact, /usr/bin/gdb on osx is a shell script that unsets DYLD_* variables before the dynamic linker even gets a chance to complain about them.

comment:40 Changed 7 years ago by jdemeyer

So you're saying that the gdb spkg wouldn't work on OS X?

comment:41 Changed 7 years ago by vbraun

I think it'll work if you make gdb suid root, or if you codesign it and make it setgid procmod. But without elevated permissions it won't:

(sage-sh) vbraun@bsd:sage-5.4.rc1$ gdb python
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin10.8.0".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /Users/vbraun/sage-5.4.rc1/local/bin/python...
warning: `/Users/vbraun/sage-5.4.rc1/spkg/build/python-2.7.3.p0/src/Modules/python.o': can't open to read symbols: No such file or directory.
(no debugging symbols found)...done.
(gdb) run
Starting program: /Users/vbraun/sage-5.4.rc1/local/bin/python 
Unable to find Mach task port for process-id 27416: (os/kern) failure (0x5).
 (please check gdb is codesigned - see taskgated(8))

Changed 7 years ago by jdemeyer

Updated patch

comment:42 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_review to positive_review

I merged both sagelib patches and removed the unneeded change to sage/tests/interrupt.pyx.

comment:43 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.6 to sage-5.7

comment:44 Changed 7 years ago by jdemeyer

  • Status changed from positive_review to needs_work

This is an annoying problem on OS X:

$ gdb
GNU gdb 6.3.50-20050815 (Apple version gdb-1822) (Sun Aug  5 03:00:42 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".
(gdb) attach 71798
We need authorization from an admin user to run the debugger.
This will only happen once per login session.
Admin username (dehayebuildbot):

At this point, gdb just "hangs" causing doctest timeouts. The best solution might simply be to not run gdb at all on OS X, since it's never going to work anyway.

Changed 7 years ago by vbraun

Initial patch

comment:45 Changed 7 years ago by vbraun

  • Description modified (diff)
  • Status changed from needs_work to needs_review

fixed

comment:46 Changed 7 years ago by jdemeyer

See http://trac.sagemath.org/13889 for how Apple screwed this up

:-)

Going to test it...

comment:47 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:48 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.7.beta0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:49 follow-up: Changed 7 years ago by jdemeyer

I noticed that the "Cython backtrace" is ordered differently from the "Stack backtrace", which is quite confusing. It would be better to respect the gdb ordering (print newest frame first).

Would it be possible to change this?

comment:50 in reply to: ↑ 49 Changed 7 years ago by vbraun

Replying to jdemeyer:

I noticed that the "Cython backtrace" is ordered differently from the "Stack backtrace"

This is now #14030

Note: See TracTickets for help on using tickets.