When creating a Linux application in a native language, such as C or C++, it's obvious that eventually a bug will appear. Some are easy to fix, but sometimes they're not. Normally you could just run gdb to figure out how to fix it, but what about a situation when the program runs smoothly under gdb, but crashes when running without any debugger?

There is one feature that can be used to troubleshoot the problem, and its name is: core dumps.

A core dump is a special file that is created when the process crashes. It's stored in binary format, but with tools like gdb it's not hard to analyze it. In fact, you don't need to analyze it at all -- it simply stores the current crashed environment of the process in such a way, that gdb can be used normally, like in every other debugging session. The only difference is that you won't be able to execute anything from such crashed process.

You can enable core dump file generation by using the ulimit command:

$ ulimit -c unlimited

This will enable core dump generation for crashed applications invoked from the current shell:

$ ./crash
Segmentation fault

$ ls core*
core.2123

2123 is the process id of the crashed application. You can load this core dump to gdb by providing its name in the argument list for gdb:

$ gdb crash core.2123

Gdb will notify about loading the file with messages like:

$ gdb crash core.2123
....
Core was generated by `./crash'.
....

You can use standard bt command to build the offending stacktrace:

(gdb) bt
#0  0x00007f8955ba4d67 in raise () from /usr/lib/libc.so.6
#1  0x00007f8955ba6118 in abort () from /usr/lib/libc.so.6
#2  0x00007f8955be4f93 in __libc_message () from /usr/lib/libc.so.6
#3  0x00007f8955bea88e in malloc_printerr () from /usr/lib/libc.so.6
#4  0x00007f8955beb04b in _int_free () from /usr/lib/libc.so.6
#5  0x00007f89564fdacc in std::string::assign(std::string const&) () from /usr/lib/libstdc++.so.6
#6  0x000000000044b551 in DataVisualizer::setCaptionText (this=0x1b831f0, s="ls") at /home/antek/workspace/mdhexedit/lib/plugins/interfaces/DataVisualizer.h:24
#7  0x0000000000448eda in MainInstance::getFileContextForNewFile (this=0x1aea058, captionText="ls") at ../src/ui/MainInstance.cpp:138
#8  0x0000000000432bb6 in FileMenu::loadFile (this=0x1af4e00, fileName="/bin/ls") at ../src/ui/menus/FileMenu.cpp:78

Way better than printf-debugging, don't you think?

I did everything from above, and I'm still not getting any coredumps!

Are you using systemd? It probably depends on the distribution, but sometimes systemd eats all the coredumps and stores them inside its own journal.

You can test if this is the case for your system by issuing a simple command:

$ coredumpctl

For example, on my home system, the output of coredumpctl looks something like this:

pon 2014-06-30 20:21:59 CEST   6705  1000   100  11 /personal/antek/.local/share/Steam/SteamApps/common/dota 2 beta/dota_linux
wto 2014-07-01 15:49:17 CEST   1660     0     0   6 /usr/bin/python3.4
wto 2014-07-01 21:50:30 CEST  18408  1000   100   6 /home/antek/.local/share/Steam/SteamApps/common/Sid Meier's Civilization V/Civ5XP
śro 2014-07-02 15:38:49 CEST   1422     0     0   6 /usr/bin/python3.4
czw 2014-07-03 10:05:46 CEST   1410     0     0   6 /usr/bin/python3.4
czw 2014-07-03 12:06:45 CEST   2515  1000   100  11 /home/antek/dev/mtf/mtf
czw 2014-07-03 12:07:15 CEST   2538  1000   100  11 /home/antek/dev/mtf/mtf

You can extract the coredump you're interested in by using coredumpctl. For example, to extract the crash from 2014-07-03 12:07:15, you can simply invoke:

$ coredumpctl dump 2538 -o core.out

This will dump this coredump directly into the core.out file. But, consider using the coredumpctl gdb command as well.