Backtraces

English | Español

Introduction

Well, not hard to guess, this is about backtraces.

Usually, it should not be necessary for the normal user to do this. However, we might have a bad day and release a somewhat buggy version or you are running SVN which can also be unstable sometimes. This is where the backtraces come in: if aMule crashes, and you get an "OOPS - aMule crashed" and so on, we'd like to know. The backtrace aMule provides is not always very usefull as it contains little information, but, as usual, there's a better way: A *real* backtrace.

The GNU Debugger

First of all, you need the GNU Debugger installed. It's called gdb and you could check for that by typing which gdb in a console window. You should see something like this:

$ which gdb
/usr/bin/gdb

If you don't have GDB installed, you will get a message like this:

$ which gdb
which: no gdb in (/bin:/usr/bin:[etc])

If that is the case, the GNU Debugger is most likely not installed on your system and you should install it before you proceed.

If your OS is Gentoo Linux you have just to type this:

# emerge -av gdb

Compiling aMule

Then, compile aMule with debugging information:

$ ./configure --enable-debug --disable-optimize --prefix=/where/to/install/aMule
$ make
$ make install

If you do not want to overwrite you old copy of aMule, simply do this instead:

$ ./configure --enable-debug --disable-optimize
$ make

aMule can then be run by going into the dir src and typing ./amule

If you are unable or unwilling to recompile, or are running a RPM version, proceed anyway, but be aware that backtraces from debugging enabled builds are much more useful to us.

On Gentoo

You must follow How to get meaningful backtraces in Gentoo.

In short, most gentoo users want to edit /etc/make.conf, add -ggdb into their CFLAGS and

FEATURES="splitdebug"

Those are needed by gdb in order to find the debug symbols. Emerge amule as usual with

USE="debug" emerge amule

If you don't use USE="debug", you will get a little bit less information from gdb, so it is better to use it. After merging amule, you can safely remove -ggdb and splitdebug from /etc/make.conf.

Create a backtrace

Now create in your home directory the file .gdbinit and put these lines into it (or you can type them in at the (gdb) prompt later):

ha SIGPIPE nostop noprint pass
ha SIG32 nostop noprint pass
ha SIG33 nostop noprint pass
ha SIG34 nostop noprint pass

For those who want to know the meaning of the previous lines: the first one avoid GDB stopping at broken pipes; the second one avoid GDB stopping at new thread.

To create a backtrace, open a console and do the following:

$ gdb /where/to/install/aMule/bin/amule
(gdb) run

Now use aMule normally until it crashes. If it crashes do the following:

(gdb) bt
(gdb) bt full
(gdb) thread apply all bt

Post the output of the last three commands in the backtraces forum with some additional comment about the circumstances the segfault happened and what aMule version you used (or checkout time for CVS).

The core file

If your aMule executable has been compiled with debug information (--enable-debug configure flag), but you were not running it from within GDB, there is still a way to generate a backtrace, if your system was configured to generate core files.

Core files are the full memory image of a process that crashed. Your session must be properly configured, so that the system generates core files. Add the following command to ~/.bashrc:

ulimit -c unlimited

Now, when a program crashes, suppose it generate the file core.1234 (this name can be different, but usually will start with 'core') you can enter GDB like that:

$ gdb --core=/path/to/amule /path/to/core/file/core.1234

REMARK: $ gdb /path/to/amule --core=/path/to/core/file/core.<pid>

and then proceed as in the last session and issue 'bt' and 'bt full'.

So, that's it, have fun with aMule

Greetings, Citroklar & Phoenix

(Most of the above shamelessly stolen from pure_ascii's post in backtraces forum, thanks, pure!)

Please read this to learn more about GDB and Valgrind.

General pitfalls and caveats

-fomit-frame-pointer

On x86 platforms the -fomit-frame-pointer compiler flag usually results in an extra free register to use, but unfortunately it most likely causes gdb to be unable to parse the resulting executable. Check your CFLAGS and CXXFLAGS variables.

-fPIE

If you use the -fPIE compiler flag to compile aMule, you must use -nopie for the linking stage: add -nopie to your LDFLAGS.

Stripping binaries

You must not strip the binaries (remove debug info), if you want to create a backtrace. The make process does not automatically strip the binaries, except for the install-strip target. Do not use it.

The GNU linker (ld) is capable of automatically stripping the binaries, if the -s or -S flag is passed to it. This is usually passed via LDFLAGS as -Wl,-s or -Wl,-S. Please check that you don't use any of them.

Examples

#0  0x000000000057b790 in ?? ()
#1  0x000000000051e66b in ?? ()
#2  0x000000000051edb6 in ?? ()
...

This is an exmple of an unuseful backtrace. It doesn't show where the error happened and what went wrong. If your backtrace looks like this, it is pretty unuseful for us. This backtrace is either created from a binary without debug information (stripped), or gdb was unable to parse it (see above section).

#0  0x1003f604 in CUpDownClient::ClearDownloadBlockRequests ()
#1  0x10044978 in CUpDownClient::Disconnected ()
#2  0x1004d958 in CClientList::ProcessDirectCallbackList ()
...

Now this is better, but still not enough. We now have global symbols, so we can see which function called which. But this still lacks line number and local symbol information, making it really hard to find the cause of the crash.

#0  0x000000000046fcab in CUpDownClient::ClearDownloadBlockRequests (this=0x45bf9e0) at BaseClient.cpp:1175
#1  0x00000000004d1480 in CUpDownClient::SetDownloadState (this=0x45bf9e0, byNewState=1 '\001') at DownloadClient.cpp:541
#2  0x00000000004703bd in CUpDownClient::Disconnected (this=0x45bf9e0, strReason=@0x7ffffc74e2b0, bFromSocket=false) at BaseClient.cpp:1239
...

Well, this is what I call a useful backtrace. It has both global and local symbols and line numbers. Now we can see the program state as it was at the time of the crash, and can possibly "easily" identify, what went wrong.

Links

• How to get meaningful backtraces in Gentoo