Difference between revisions of "Backtraces"

From AMule Project FAQ
Jump to: navigation, search
m
(Backtraces in Fedora : Using ABRT)
 
(16 intermediate revisions by 10 users not shown)
Line 1: Line 1:
 +
<center>
 +
[[Backtraces|English]] |
 +
[[Backtraces-es|Espa&ntilde;ol]]
 +
</center>
 +
 
== Introduction ==
 
== Introduction ==
 
 
Well, not hard to guess, this is about backtraces.
 
Well, not hard to guess, this is about backtraces.
  
Usually, it should not be necessary for the normal user to do this. However, we might have a bad day and release a somewhat buggy version or you are running [http://www.gnu.org/software/cvs CVS] which can also be unstable sometimes.
+
Usually, it should not be necessary for the normal user to do this. However, we might have a bad day and release a somewhat buggy version or you are running [http://subversion.tigris.org SVN] which can also be unstable sometimes.
 
This is where the backtraces come in: if [[aMule]] crashes, and you get an "OOPS - aMule crashed" and so on, we'd like to know. The backtrace [[aMule]] provides is not always very usefull as it contains little information, but, as usual, there's a better way: A *real* backtrace.
 
This is where the backtraces come in: if [[aMule]] crashes, and you get an "OOPS - aMule crashed" and so on, we'd like to know. The backtrace [[aMule]] provides is not always very usefull as it contains little information, but, as usual, there's a better way: A *real* backtrace.
  
 
== The GNU Debugger ==
 
== The GNU Debugger ==
 
 
First of all, you need the [http://www.gnu.org/software/gdb/gdb.html GNU Debugger] installed. It's called ''gdb'' and you could check for that by typing ''which gdb'' in a console window. You should see something like this:
 
First of all, you need the [http://www.gnu.org/software/gdb/gdb.html GNU Debugger] installed. It's called ''gdb'' and you could check for that by typing ''which gdb'' in a console window. You should see something like this:
  
''<pre>
+
$ which gdb
$ which gdb
+
/usr/bin/gdb
/usr/bin/gdb
+
</pre>''
+
  
 
If you don't have [http://www.gnu.org/software/gdb/gdb.html GDB] installed, you will get a message like this:<br>
 
If you don't have [http://www.gnu.org/software/gdb/gdb.html GDB] installed, you will get a message like this:<br>
  
''<pre>
+
$ which gdb
$ which gdb
+
which: no gdb in (/bin:/usr/bin:[etc])
which: no gdb in (/bin:/usr/bin:[etc])
+
</pre>''
+
  
 
If that is the case, the [http://www.gnu.org/software/gdb/gdb.html GNU Debugger] is most likely not installed on your system and you should install it before you proceed.
 
If that is the case, the [http://www.gnu.org/software/gdb/gdb.html GNU Debugger] is most likely not installed on your system and you should install it before you proceed.
  
 
If your OS is [http://www.gentoo.org Gentoo Linux] you have just to type this:
 
If your OS is [http://www.gentoo.org Gentoo Linux] you have just to type this:
''<pre>
+
# emerge -av gdb
# emerge -av gdb
+
</pre>''
+
  
 
== Compiling [[aMule]] ==
 
== Compiling [[aMule]] ==
 
 
Then, compile [[aMule]] with debugging information:
 
Then, compile [[aMule]] with debugging information:
  
''<pre>
+
$ ./configure --enable-debug --disable-optimize --prefix=/where/to/install/aMule
$ ./configure --enable-debug --disable-optimize --prefix=/where/to/install/aMule
+
$ make
$ make
+
$ make install
$ make install
+
</pre>''
+
  
 
If you do not want to overwrite you old copy of [[aMule]], simply do this instead:
 
If you do not want to overwrite you old copy of [[aMule]], simply do this instead:
  
''<pre>
+
$ ./configure --enable-debug --disable-optimize
$ ./configure --enable-debug --disable-optimize
+
$ make
$ make
+
</pre>''
+
  
 
[[aMule]] can then be run by going into the dir ''src'' and typing ''./amule''
 
[[aMule]] can then be run by going into the dir ''src'' and typing ''./amule''
Line 50: Line 42:
 
If you are unable or unwilling to recompile, or are running a RPM version, proceed anyway, but be aware that backtraces from debugging enabled builds are much more useful to us.
 
If you are unable or unwilling to recompile, or are running a RPM version, proceed anyway, but be aware that backtraces from debugging enabled builds are much more useful to us.
  
 +
=== On Gentoo ===
 +
You must follow [http://www.gentoo.org/proj/en/qa/backtraces.xml How to get meaningful backtraces in Gentoo].
  
== Create a backtrace ==
+
In short, most gentoo users want to edit ''/etc/make.conf'', add ''-ggdb'' into their CFLAGS and
 +
FEATURES="splitdebug"
 +
Those are needed by gdb in order to find the debug symbols.
 +
Emerge amule as usual with
 +
USE="debug" emerge amule
 +
If you don't use ''USE="debug"'', you will get a little bit less information from gdb, so it is better to use it.
 +
After merging amule, you can safely remove ''-ggdb'' and ''splitdebug'' from ''/etc/make.conf''.
  
 +
== Create a backtrace ==
 
Now create in your home directory the file ''.gdbinit'' and put these lines into it (or you can type them in at the ''(gdb)'' prompt later):
 
Now create in your home directory the file ''.gdbinit'' and put these lines into it (or you can type them in at the ''(gdb)'' prompt later):
''<pre>
+
 
ha SIGPIPE nostop noprint pass
+
ha SIGPIPE nostop noprint pass
ha SIG32 nostop noprint pass
+
ha SIG32 nostop noprint pass
ha SIG33 nostop noprint pass
+
ha SIG33 nostop noprint pass
ha SIG34 nostop noprint pass
+
ha SIG34 nostop noprint pass
</pre>''
+
  
 
For those who want to know the meaning of the previous lines:
 
For those who want to know the meaning of the previous lines:
Line 67: Line 67:
 
To create a backtrace, open a console and do the following:
 
To create a backtrace, open a console and do the following:
  
''<pre>
+
$ gdb /where/to/install/aMule/bin/amule
$ gdb /where/to/install/aMule/bin/amule
+
(gdb) run
(gdb) run
+
</pre>''
+
  
 
Now use [[aMule]] normally until it crashes. If it crashes do the following:
 
Now use [[aMule]] normally until it crashes. If it crashes do the following:
  
''<pre>
+
(gdb) bt
(gdb) bt
+
(gdb) bt full
(gdb) bt full
+
(gdb) thread apply all bt
(gdb) thread apply all bt
+
</pre>''
+
  
Post the output of the last three commands in the [http://www.amule.org/amule/board.php?boardid=33 backtraces forum] with some additional comment about the circumstances the segfault happened and what [[aMule]] version you used (or checkout time for [http://www.gnu.org/software/cvs CVS]).
+
Post the output of the last three commands in the [http://forum.amule.org/index.php?board=33.0 backtraces forum] with some additional comment about the circumstances the segfault happened and what [[aMule]] version you used (or checkout time for [http://www.gnu.org/software/cvs CVS]).
  
 
== The core file ==
 
== The core file ==
 
 
If your [[aMule]] executable has been compiled with debug information (''--enable-debug'' configure flag), but you were not running it from within [http://www.gnu.org/software/gdb/gdb.html GDB], there is still a way to generate a backtrace, if your system was configured to generate '''core files'''.
 
If your [[aMule]] executable has been compiled with debug information (''--enable-debug'' configure flag), but you were not running it from within [http://www.gnu.org/software/gdb/gdb.html GDB], there is still a way to generate a backtrace, if your system was configured to generate '''core files'''.
  
Line 92: Line 87:
 
Now, when a program crashes, suppose it generate the file ''core.1234'' (this name can be different, but usually will start with 'core') you can enter [http://www.gnu.org/software/gdb/gdb.html GDB] like that:
 
Now, when a program crashes, suppose it generate the file ''core.1234'' (this name can be different, but usually will start with 'core') you can enter [http://www.gnu.org/software/gdb/gdb.html GDB] like that:
  
''$ gdb /path/to/amule /path/to/core/file/core.1234''
+
''$ gdb --core=/path/to/amule /path/to/core/file/core.1234''
 +
 
 +
REMARK: ''$ gdb /path/to/amule --core=/path/to/core/file/core.<pid>''
  
 
and then proceed as in the last session and issue 'bt' and 'bt full'.
 
and then proceed as in the last session and issue 'bt' and 'bt full'.
Line 103: Line 100:
  
 
Please read [[Using_gdb_and_valgrind|this]] to learn more about [http://www.gnu.org/software/gdb/gdb.html GDB] and [http://valgrind.kde.org Valgrind].
 
Please read [[Using_gdb_and_valgrind|this]] to learn more about [http://www.gnu.org/software/gdb/gdb.html GDB] and [http://valgrind.kde.org Valgrind].
 +
 +
== General pitfalls and caveats ==
 +
 +
;<tt>-fomit-frame-pointer</tt>
 +
:On <tt>x86</tt> platforms the <tt>-fomit-frame-pointer</tt> compiler flag usually results in an extra free register to use, but unfortunately it most likely causes ''gdb'' to be unable to parse the resulting executable. Check your <tt>CFLAGS</tt> and <tt>CXXFLAGS</tt> variables.
 +
 +
;<tt>-fPIE</tt>
 +
:If you use the <tt>-fPIE</tt> compiler flag to compile aMule, you must use <tt>-nopie</tt> for the linking stage: add <tt>-nopie</tt> to your <tt>LDFLAGS</tt>.
 +
 +
;Stripping binaries
 +
:You must not strip the binaries (remove debug info), if you want to create a backtrace. The make process does not automatically strip the binaries, except for the <tt>install-strip</tt> target. Do not use it.
 +
:The GNU linker (<tt>ld</tt>) is capable of automatically stripping the binaries, if the <tt>-s</tt> or <tt>-S</tt> flag is passed to it. This is usually passed via <tt>LDFLAGS</tt> as <tt>-Wl,-s</tt> or <tt>-Wl,-S</tt>. Please check that you don't use any of them.
 +
 +
== Examples ==
 +
 +
#0  0x000000000057b790 in ?? ()
 +
#1  0x000000000051e66b in ?? ()
 +
#2  0x000000000051edb6 in ?? ()
 +
...
 +
 +
This is an exmple of an unuseful backtrace. It doesn't show where the error happened and what went wrong. If your backtrace looks like this, it is pretty unuseful for us. This backtrace is either created from a binary without debug information (stripped), or ''gdb'' was unable to parse it (see above section).
 +
 +
#0  0x1003f604 in CUpDownClient::ClearDownloadBlockRequests ()
 +
#1  0x10044978 in CUpDownClient::Disconnected ()
 +
#2  0x1004d958 in CClientList::ProcessDirectCallbackList ()
 +
...
 +
 +
Now this is better, but still not enough. We now have global symbols, so we can see which function called which. But this still lacks line number and local symbol information, making it really hard to find the cause of the crash.
 +
 +
#0  0x000000000046fcab in CUpDownClient::ClearDownloadBlockRequests (this=0x45bf9e0) at BaseClient.cpp:1175
 +
#1  0x00000000004d1480 in CUpDownClient::SetDownloadState (this=0x45bf9e0, byNewState=1 '\001') at DownloadClient.cpp:541
 +
#2  0x00000000004703bd in CUpDownClient::Disconnected (this=0x45bf9e0, strReason=@0x7ffffc74e2b0, bFromSocket=false) at BaseClient.cpp:1239
 +
...
 +
 +
Well, this is what I call a useful backtrace. It has both global and local symbols and line numbers. Now we can see the program state as it was at the time of the crash, and can possibly "easily" identify, what went wrong.
 +
 +
== Backtraces in Fedora : Using ABRT ==
 +
 +
Fedora includes an "Automatic Bug Reporting Tool" which helps in creating backtraces.
 +
 +
ABRT can be used to provide useful amule backtraces on the following conditions:
 +
 +
* you are running Fedora 14 or newer
 +
 +
* amule is installed from rpmfusion, or you have compiled yourself amule with debug information (see above), using Fedora wxGTK libraries
 +
 +
* for self-made amule binaries you have configured "OpenGPGCheck = no" in /etc/abrt/abrt.conf
 +
 +
 +
Please use the "Local GNU Debugger" as analyzer choice, and the "Logger" facility to create a backtrace report in a file you can send to the aMule team.
 +
 +
 +
You can find information on ABRT usage at the following link:
 +
 +
* [https://fedorahosted.org/abrt/wiki/AbrtBasicFunctionality ABRT Basic functionality]
 +
 +
== Links ==
 +
* [http://www.gentoo.org/proj/en/qa/backtraces.xml How to get meaningful backtraces in Gentoo]

Latest revision as of 13:49, 21 June 2011

English | Español

Introduction

Well, not hard to guess, this is about backtraces.

Usually, it should not be necessary for the normal user to do this. However, we might have a bad day and release a somewhat buggy version or you are running SVN which can also be unstable sometimes. This is where the backtraces come in: if aMule crashes, and you get an "OOPS - aMule crashed" and so on, we'd like to know. The backtrace aMule provides is not always very usefull as it contains little information, but, as usual, there's a better way: A *real* backtrace.

The GNU Debugger

First of all, you need the GNU Debugger installed. It's called gdb and you could check for that by typing which gdb in a console window. You should see something like this:

$ which gdb
/usr/bin/gdb

If you don't have GDB installed, you will get a message like this:

$ which gdb
which: no gdb in (/bin:/usr/bin:[etc])

If that is the case, the GNU Debugger is most likely not installed on your system and you should install it before you proceed.

If your OS is Gentoo Linux you have just to type this:

# emerge -av gdb

Compiling aMule

Then, compile aMule with debugging information:

$ ./configure --enable-debug --disable-optimize --prefix=/where/to/install/aMule
$ make
$ make install

If you do not want to overwrite you old copy of aMule, simply do this instead:

$ ./configure --enable-debug --disable-optimize
$ make

aMule can then be run by going into the dir src and typing ./amule

If you are unable or unwilling to recompile, or are running a RPM version, proceed anyway, but be aware that backtraces from debugging enabled builds are much more useful to us.

On Gentoo

You must follow How to get meaningful backtraces in Gentoo.

In short, most gentoo users want to edit /etc/make.conf, add -ggdb into their CFLAGS and

FEATURES="splitdebug"

Those are needed by gdb in order to find the debug symbols. Emerge amule as usual with

USE="debug" emerge amule

If you don't use USE="debug", you will get a little bit less information from gdb, so it is better to use it. After merging amule, you can safely remove -ggdb and splitdebug from /etc/make.conf.

Create a backtrace

Now create in your home directory the file .gdbinit and put these lines into it (or you can type them in at the (gdb) prompt later):

ha SIGPIPE nostop noprint pass
ha SIG32 nostop noprint pass
ha SIG33 nostop noprint pass
ha SIG34 nostop noprint pass

For those who want to know the meaning of the previous lines: the first one avoid GDB stopping at broken pipes; the second one avoid GDB stopping at new thread.

To create a backtrace, open a console and do the following:

$ gdb /where/to/install/aMule/bin/amule
(gdb) run

Now use aMule normally until it crashes. If it crashes do the following:

(gdb) bt
(gdb) bt full
(gdb) thread apply all bt

Post the output of the last three commands in the backtraces forum with some additional comment about the circumstances the segfault happened and what aMule version you used (or checkout time for CVS).

The core file

If your aMule executable has been compiled with debug information (--enable-debug configure flag), but you were not running it from within GDB, there is still a way to generate a backtrace, if your system was configured to generate core files.

Core files are the full memory image of a process that crashed. Your session must be properly configured, so that the system generates core files. Add the following command to ~/.bashrc:

ulimit -c unlimited

Now, when a program crashes, suppose it generate the file core.1234 (this name can be different, but usually will start with 'core') you can enter GDB like that:

$ gdb --core=/path/to/amule /path/to/core/file/core.1234

REMARK: $ gdb /path/to/amule --core=/path/to/core/file/core.<pid>

and then proceed as in the last session and issue 'bt' and 'bt full'.

So, that's it, have fun with aMule

Greetings, Citroklar & Phoenix

(Most of the above shamelessly stolen from pure_ascii's post in backtraces forum, thanks, pure!)

Please read this to learn more about GDB and Valgrind.

General pitfalls and caveats

-fomit-frame-pointer
On x86 platforms the -fomit-frame-pointer compiler flag usually results in an extra free register to use, but unfortunately it most likely causes gdb to be unable to parse the resulting executable. Check your CFLAGS and CXXFLAGS variables.
-fPIE
If you use the -fPIE compiler flag to compile aMule, you must use -nopie for the linking stage: add -nopie to your LDFLAGS.
Stripping binaries
You must not strip the binaries (remove debug info), if you want to create a backtrace. The make process does not automatically strip the binaries, except for the install-strip target. Do not use it.
The GNU linker (ld) is capable of automatically stripping the binaries, if the -s or -S flag is passed to it. This is usually passed via LDFLAGS as -Wl,-s or -Wl,-S. Please check that you don't use any of them.

Examples

#0  0x000000000057b790 in ?? ()
#1  0x000000000051e66b in ?? ()
#2  0x000000000051edb6 in ?? ()
...

This is an exmple of an unuseful backtrace. It doesn't show where the error happened and what went wrong. If your backtrace looks like this, it is pretty unuseful for us. This backtrace is either created from a binary without debug information (stripped), or gdb was unable to parse it (see above section).

#0  0x1003f604 in CUpDownClient::ClearDownloadBlockRequests ()
#1  0x10044978 in CUpDownClient::Disconnected ()
#2  0x1004d958 in CClientList::ProcessDirectCallbackList ()
...

Now this is better, but still not enough. We now have global symbols, so we can see which function called which. But this still lacks line number and local symbol information, making it really hard to find the cause of the crash.

#0  0x000000000046fcab in CUpDownClient::ClearDownloadBlockRequests (this=0x45bf9e0) at BaseClient.cpp:1175
#1  0x00000000004d1480 in CUpDownClient::SetDownloadState (this=0x45bf9e0, byNewState=1 '\001') at DownloadClient.cpp:541
#2  0x00000000004703bd in CUpDownClient::Disconnected (this=0x45bf9e0, strReason=@0x7ffffc74e2b0, bFromSocket=false) at BaseClient.cpp:1239
...

Well, this is what I call a useful backtrace. It has both global and local symbols and line numbers. Now we can see the program state as it was at the time of the crash, and can possibly "easily" identify, what went wrong.

Backtraces in Fedora : Using ABRT

Fedora includes an "Automatic Bug Reporting Tool" which helps in creating backtraces.

ABRT can be used to provide useful amule backtraces on the following conditions:

  • you are running Fedora 14 or newer
  • amule is installed from rpmfusion, or you have compiled yourself amule with debug information (see above), using Fedora wxGTK libraries
  • for self-made amule binaries you have configured "OpenGPGCheck = no" in /etc/abrt/abrt.conf


Please use the "Local GNU Debugger" as analyzer choice, and the "Logger" facility to create a backtrace report in a file you can send to the aMule team.


You can find information on ABRT usage at the following link:

Links