Automatic Backtraces

Posted: September 6th, 2009 | Author: | Filed under: Debugging, macOS, Programming | No Comments »

Here are different solutions to display automatically a backtrace when entering a specific function in your application. As an example we will take the following program. It’s a really simple program: the main function calls the function “function1” which prints a string.

#include <stdio.h>

void function1()
{
	fprintf(stderr, "Who called this function?\n");
}

int main()
{
	function1();
	return 0;
}

1- With GDB

The first solution is to set a breakpoint to the function “function1” and run your program in the debugger. Each time GDB breaks you can inspect the backtrace and see why your function is called.

You can extend this method and automatically dump a backtrace each time the function is called by creating a command on a breakpoint in GDB. You can indeed give any breakpoint a series of commands to execute when your program stops due to that breakpoint.

Following is a simple GDB script that can be loaded using the source command in GDB:

# We create a breakpoint using "future break"  
# If we break due to this breakpoint, the command  
#following will be executed.  
fb function1  
command  
   backtrace
   continue  
end 

For such a script it is faster to manually type the script in GDB with some abbreviations (“c” for continue, “bt” for backtrace):

(gdb) fb function1
Breakpoint 1 at 0x100000eac
(gdb) command
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just “end”.
>bt
>c
>end
(gdb) r
Starting program: a.out
Reading symbols for shared libraries +. done

Breakpoint 1, 0x0000000100000eac in function1 ()
#0 0x0000000100000eac in function1 ()
#1 0x0000000100000edc in main ()
Who called this function?

Program exited normally.
(gdb)

2- Without GDB

The solution with GDB is powerful and requires no code change but it is slow (you need to run in the debugger). It is possible to display the backtraces without GDB by using the backtrace() and backtrace_symbols() functions available in 10.5 and later. Here is a sample code to show how to use these functions:

#include <execinfo.h>
#include <stdio.h>

#define BACKTRACE_MAX_DEPTH 20

static void dumpBacktrace()
{
	void *stack_frame[BACKTRACE_MAX_DEPTH];

	// backtrace() writes the function return addresses of the current
	// call stack to the array of pointers referenced by array.
	// At most, size pointers are written.
	// The number of pointers actually written to array is returned.

	int number_of_frames = backtrace(stack_frame, BACKTRACE_MAX_DEPTH);

	// backtrace_symbols() attempts to transform a call stack obtained
	// by backtrace() into an array of human readable strings using
	// dladdr(). The array of strings returned has size elements.
	// It is allocated using malloc() and should be released using free().
	// There is no need to free the individual strings in the array.

	char** human_readable_frames =
               backtrace_symbols(stack_frame, number_of_frames);

	fprintf(stderr, "--------------------\nDumpBacktrace:\n");
	int i = 0;
	for (i = 0; i < number_of_frames; i++)
	{
		fprintf(stderr, "\t %s\n", human_readable_frames[i]);
	}

	// This needs to be released
	free(human_readable_frames);
}

void function1()
{
	fprintf(stderr, "Who called this function?\n");
	dumpBacktrace();
}

int main()
{
	function1();
	return 0;
}

Here is the output:

Output


getrlimit and randomness

Posted: August 6th, 2009 | Author: | Filed under: Debugging, macOS, Programming | 2 Comments »

To determine the maximum number of files your application can open, there is a function getrlimit() available. Here is what tells us the man page:

int getrlimit(int resource, struct rlimit *rlp);

A resource limit is specified as a soft limit and a hard limit. When a soft limit is exceeded a process may receive a signal (for example, if the cpu time or file size is exceeded), but it will be allowed to continue execution until it reaches the hard limit (or modifies its resource limit). The rlimit structure  is used to specify the hard and soft limits on a resource,

struct rlimit {

rlim_t  rlim_cur;       /* current (soft) limit */
rlim_t  rlim_max;       /* hard limit */

};

Let’s create an application to print the soft and hard limit. I simply created a new Cocoa project and add this awakeFromNib function:

#include <stdio.h>
#include <sys/resource.h>

-(void)awakeFromNib
{
     struct rlimit lim;
     int theErr = getrlimit(RLIMIT_NOFILE, &lim);
     if(theErr == 0)
     {
        if(lim.rlim_max == RLIM_INFINITY)
           fprintf(stderr, "soft: %llu ; hard: RLIM_INFINITYn", lim.rlim_cur);
        else
            fprintf(stderr, "soft: %llu ; hard: %llun", lim.rlim_cur, lim.rlim_max);
     }
     else
     {
         fprintf(stderr, "getrlimit returned an error %dn", theErr);
     }
}

What would you expect? 256? 512? 1024? More?
Well the value depends on how you run the application! Here are the results on 10.5.8 i386:

– If you run the application by double clicking on the icon in the Finder:

soft: 256 ; hard: RLIM_INFINITY

– If you run the application from the terminal:

soft: 256 ; hard: RLIM_INFINITY

– If you run the application from Xcode without gdb:

soft: 2560 ; hard: RLIM_INFINITY

– If you run the application from Xcode with gdb:

soft: 10240 ; hard: RLIM_INFINITY

– If you run the application from the Terminal in gdb:

soft: 10240 ; hard: RLIM_INFINITY

– If you run a slightly modified application from the Finder and attach it to gdb and call manually the function:

soft: 256 ; hard: RLIM_INFINITY

This is quite confusing! I made a bug report: radar://7123983


Major issue in SimplePing

Posted: November 20th, 2008 | Author: | Filed under: Debugging, Programming | Tags: | 3 Comments »

The SimplePing sample from Apple can fail to perform a ping request (ICMP).

SimplePing is a small example to show how to use the Internet Control Message Protocol (ICMP) “ECHO” facility, measure round-trip-delays and packet loss across network paths.
The code is relatively simple and contains a main function ‘int SimplePing(…)’ which pings a given remote host with a given number of packets and with a given timeout to wait for responses on each packet.

There is yet a major issue in that code.

If you look at the function which creates the ICMP packet ‘CreateAndSendICMPPacket(…)’ you should see that the pid of the application is stored in the icmp_id field (identifier) of the ICMP packet:

CreateAndSendICMPPacket in SimplePing

This is used to know later that the ping originated from our application (here SimplePing). If you look at the function ‘WaitAndPrintICMPs(…)’ which waits for the answer, you see:

WaitAndPrintICMPs in SimplePing

When you first read that code, it seems to be right. There is yet a major issue.

getpid() is delared as pid_t getpid(void); (see unistd.h) and thus returns a 32 bits value. However the field icmp_id of the icmp header structure (see ip_icmp.h) is a n_short which can only contains 16 bits:

icmp header structure

As a consequence, if the pid of the SimplePing application is bigger than 0xFFFF (65535), the pings will fail.

If you search on google “icmp_id getpid”, you will see a lot of souce code containing the exact same issue, even in Apple source code.

If your application contains such code, verify that you store correctly only 16 bits of the pid in the icmp_id field:

icmpHeader.icmp_id = getpid() & 0xFFFF;

and that you compare only 16 bits of the ping answer:

if (icmpPacket->icmpHeader.icmp_id == (getpid() & 0xFFFF))

Note:
A bug report has been made to Apple: rdar://6385021.


Programmatically break into GDB

Posted: August 24th, 2008 | Author: | Filed under: Debugging | Tags: , , , | 4 Comments »

You may want to stop into the debugger if something bad happened while executing your program. The easier way to do that is to have a breakpoint always set and to run your program in the debugger.

But sometimes you want to break even if there are no breakpoint. For example you know that something is wrong now and your program will crash later.

The following BreakIntoGDB() function checks if the application runs with GDB attached. If you are running your application with the debugger attached, BreakIntoGDB calls the Debugger() function. If you are running without GDB attached, BreakIntoGDB opens a Terminal window and attaches your application to the debugger using an Applescript. In both cases, your program stops into GDB and your can examine what’s wrong.

//
//  DebuggingUtilities.h
//  BreakingIntoGDB
//
//  Created by Alexandre Colucci on 23/08/08.
//

#import <Foundation/Foundation.h>
#import <ApplicationServices/ApplicationServices.h>

#include <assert.h>
#include <stdbool.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/sysctl.h>
#include <sys/proc.h>

// From Technical Q&A QA1361
// Returns true if the current process
// is being debugged (either running
// under the debugger or has a debugger
// attached post facto).
static bool AmIBeingDebugged(void)
{
    int                 junk;
    int                 mib[4];
    struct kinfo_proc   info;
    size_t              size;
	
    // Initialize the flags so that,
    // if sysctl fails for some bizarre
    // reason, we get a predictable result.
	
    info.kp_proc.p_flag = 0;
	
    // Initialize mib, which tells sysctl the info
    // we want, in this case we're looking for
    // information about a specific process ID.
	
    mib[0] = CTL_KERN;
    mib[1] = KERN_PROC;
    mib[2] = KERN_PROC_PID;
    mib[3] = getpid();
	
    // Call sysctl.
	
    size = sizeof(info);
    junk = sysctl(mib, sizeof(mib) / sizeof(*mib), &info, &size, NULL, 0);

    assert(junk == 0);
	
    // We're being debugged if the P_TRACED
    // flag is set.
	
    return ( (info.kp_proc.p_flag & P_TRACED) != 0 );
}

static void BreakIntoGDB()
{
	if(AmIBeingDebugged())
	{
		// GDB is already attached to the application.
		// We simply break using the Debugger function.
		Debugger();
	}
	else
	{
		// GDB is not running. We open a
		// Terminal window and attach the
		// current application.
			
		ProcessSerialNumber psn = {0,kCurrentProcess};
		pid_t thePID;
		OSStatus err = GetProcessPID(&psn, &thePID);
		if(err == noErr)
		{
			NSString *scriptSource =
 [NSString stringWithFormat:@"tell application \"Terminal\" ndo script \"gdb attach \" & %d nend tell", thePID];

			NSAppleScript *appleScript = [[NSAppleScript alloc] initWithSource:scriptSource];
			if(appleScript != nil)
			{
				NSDictionary *errorDict;
				
				// Don't check the returned value
				[appleScript executeAndReturnError:&errorDict];
			}

			sleep(2);
		}
	}
}

To break into the debugger, you just have to call the BreakIntoGDB() function like in the following example.

-(IBAction)breakNow:(id)sender
{
	BreakIntoGDB();
}

Several other notes:


Breakpoint conditions with GDB

Posted: August 8th, 2008 | Author: | Filed under: Debugging | Tags: , , , | 2 Comments »

When debugging your application, you use breakpoints. The program will return control to GDB every time it reaches a breakpoint you set. This may not be desirable if you have breakpoint on a method that is called many times and you want to break only with certain values passed to that method. GDB provides several ways to do conditional breakpoints that I’ll try to explain.

 

1- The problem

Let take a really simple application which calls 10 times a function.

#import <Foundation/Foundation.h>

void myFunction()
{
   // Initialize the counter to 0
   static int sFunctionCounter = 0;

   // Each time we enter the function,
   // increment the counter
   sFunctionCounter++;

   NSLog(@"function called: %d", sFunctionCounter);
}

int main (int argc, const char * argv[])
{
   NSAutoreleasePool * pool =[[NSAutoreleasePool alloc] init];
   int i;

   // Call 10 times myFunction.
   for(i = 0 ; i < 10 ; i++)
        myFunction();

   [pool release];
   return 0;
}

Now imagine that you want to break in myFunction, but only the 4th time you enter that function. One easy solution would be to modify the function by adding a if statement and setting a breakpoint accordingly.

myFunction

This method works fine in that case but you need to modify your function, you need to rebuild, you can’t do it if you don’t have the sources…

2- Conditional breakpoints

GDB can let you add breakpoints to stop whenever a certain point in the program is reached. The problem is that if you set a breakpoint to the function myFunction, you will break 10 times and will need to continue until you get the right occurrence.
Hopefully you can specify a condition (boolean expression) on your breakpoint so that your program stops only if the condition is true.

Let’s take the sample of the previously detailed program.
We will use the Console in Xcode but you can use directly GDB in command line if you prefer.

Here is what to do:
– First we set a breakpoint at the entry of the main function of the program.
– Set a breakpoint to stop on myFunction: break myFunction.
– Finally tell the debugger to only break if sFunctionCounter is equal to 4: condition 9 sFunctionCounter == 4
The program will run and will stop at the 4th call of the function myFunction as you can see on this screenshot:

Console

3- Breakpoints commands

Conditional breakpoints are useful but not really flexible. GDB provides another solution: the breakpoint commands.
You can give any breakpoint a series of commands to execute when your program stops due to that breakpoint. For example, you can display the values of certain expressions or enable/disable some breakpoints.
We will take another simple example. Our new program call 10 times 2 methods with detachNewThreadSelector. The calls to method1: and method2: are asynchronous so that we can’t tell if method1: will be executed before method2:.

#import <Foundation/Foundation.h>

@interface Test : NSObject
{
}

-(void)method1:(id)sender;
-(void)method2:(id)sender;

@end

@implementation Test

-(void)method1:(id)sender
{
   // Do something
}

-(void)method2:(id)sender
{
   // Do something
}

@end

int main (int argc, const char * argv[])
{
   NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

   int i;
   Test *myTest = [[Test alloc] init];

   // Call 10 times the methods method1: and method2:.
   // The calls are asynchronous so that we can't tell if method1:
   // will be executed before method2:.
   
   for(i = 0 ; i < 10 ; i++)
   {
      NSLog(@"iteration: %d", i);

      // Call method1: asynchronously
      [NSThread detachNewThreadSelector:@selector(method1:) toTarget:myTest withObject:nil];
   
      // Call method2: asynchronously
      [NSThread detachNewThreadSelector:@selector(method2:) toTarget:myTest withObject:nil];

      // Wait before continuing to loop
      [NSThread sleepUntilDate:[NSDate dateWithTimeIntervalSinceNow:1]];
   }

   [myTest release];

   [pool release];
   return 0;
}

What we would like to do, is to break in method2: only if method2: is called before method1:.
This is not trivial to do with conditional breakpoints, and not easy to do by modifying the source code.

Let’s talk bout the powerful breakpoint commands, which can solve this. You will need to write a little txt file. This file contains the commands to execute after a breakpoint is reached.


# Define two integers which will be incremented
# each time you enter the corresponding method.
set $method1_Counter = 0
set $method2_Counter = 0

# We create a breakpoint using "future break"
# If we break due to this breakpoint, the command
#following will be executed.
fb method1:
command
   set $method1_Counter = $method1_Counter + 1
   printf "method1 calledn"
   continue
end

# We create the second breakpoint and
# the coressponding command.
fb method2:
command
   set $method2_Counter = $method2_Counter + 1
   printf "method2 calledn"

   if($method1_Counter < $method2_Counter)
      printf "method2 was called before method1!n"
   else
      continue
   end
end

As you can see, this little txt file is really simple. We define 2 breakpoints, one in method1: and one in method2:. For each breakpoint defined, we define a command (kind of a small program) which will be executed.

The command related to the breakpoint method1: prints “method1 called” and we tell the debugger to continue the execution of the program.
The command related to the breakpoint method2: prints “method2 called” and we only tell GDB to continue the program if method1_Counter < method2_Counter (the number of calls to method2: is greater than the number of calls to method1:). If method1: was called before method2:, we continue the execution of the program.

Now that we have our txt file, we can start our application, load the txt file using the source command and see that indeed we only break if method2: is called before method1:

Console


libMallocDebug

Posted: July 19th, 2008 | Author: | Filed under: Debugging | Tags: , , , | No Comments »

Here is a little description on how to use libMallocDebug for developers and QA people. It seems not many people know about it.
libMallocDebug is really useful in the case you have a crash sometimes when doing something but it is hard to reproduce.

What is libMallocDebug?

libMallocDebug is a debugging malloc library that can track down memory related bugs and can be turned on and off at runtime. If you use this library, it will replace the malloc functions with a debug version of these functions. It contains several interesting options but only 3 are really useful.

MallocGuardEdges: This option adds guard pages to detect buffer overflows. Buffer overflows are not easy to find because it might crash randomly depending on what you write and where.

MallocScribble: This option detects writes to a freed block by setting its contents to 0x55 when deallocated. This can detect bugs like double frees, or using freed memory. Dereferencing a pointer in cleared freed memory will cause the program to reference the memory at 0x55555555, which usually is unallocated and will cause an immediate crash if the pointer is dereferenced for reading or writing.

MallocPreScribble: This option detects reads of uninitialized memory (also known as uninitialized variables) by setting its contents to 0xAA when allocated. Reading an uninitialized pointer from that memory will cause the program to reference the memory at 0xAAAAAAAA, which is usually unallocated and will cause an immediate crash if the pointer is dereferenced for reading or writing.
Using libMallocDebug (without sources) for beta testers:

You can run any application with the libMallocDebug library. To do that, open a terminal window (bash) and type:

export DYLD_INSERT_LIBRARIES=/usr/lib/libMallocDebug.A.dylib
export MallocGuardEdges=YES
export MallocScribble=YES
export MallocPreScribble=YES
/Applications/MyAppDBG.app/Contents/MacOS/MyAppDBG

This will run MyApp using libMallocDebug. You will see in the Terminal window some printf made by libMallocDebug. To see if libMallocDebug is used, the first line should be:
“libMallocDebug[MyAppDBG-10714]: initializing libMallocDebug on thread 10b”. Here is an example:

Alex:~ leopard$ export DYLD_INSERT_LIBRARIES=/usr/lib/libMallocDebug.A.dylib
Alex:~ leopard$ export MallocGuardEdges=YES
Alex:~ leopard$ export MallocScribble=YES
Alex:~ leopard$ export MallocPreScribble=YES
Alex:~ leopard$ /Applications/MyAppDBG.app/Contents/MacOS/MyAppDBG 
libMallocDebug[MyAppDBG-4388]: initializing libMallocDebug on thread 10b
task_set_exception_ports (os/kern) invalid argument
libMallocDebug[MyAppDBG-4388]: target application recently wrote to freed malloc’d memory at: 0x32b03c0, suggesting it didn’t realize the memory had been freed

Using libMallocDebug (with sources) for developers:

Select the Executable in the Project and add the 4 environment variables as shown on the following picture:

Setting environment variables in Xcode.

This will help to reproduce crashes. You can also use gdb and libMallocDebug at the same time.

Limitations and issues:

  • This tool helps to find memory bugs but it might not find all of them. 
  • It checks memory at runtime and thus only detects bugs in the features you try.
  • This library affects performances. So don’t use it while doing performance comparisons.
  • For developers: don’t forget to uncheck the environment variables in the Executable once you are done.
  • For developers: You can use gdb and libmallocDebug at the same time to reproduce and debug issues. But there is a bug on Leopard with Intel machines (rdar://5611207). You can’t use gdb in Xcode and libMallocDebug at the same time. There are several workarounds: Boot on Tiger with your intel machine, use a PowerPC machine with Leopard, or use gdb from the command line on Leopard with an intel machine.

[Update]: Thomas Clement pointed out several things.

  • MallocGuardEdges, MallocScribble and MallocPreScribble are available in the default memory allocator as mentioned in Mac OS X Debugging Magic Tech Note. That means you can use them without having to use LibMallocDebug. Of course when you use LibMallocDebug I recommend you to always enable these environment variables to detect more possible issues.
  • The application MallocDebug (available in the CHUD tools) uses LibMallocDebug. If you don’t like to set environment variables, you can use that application which is really nice.
  • There is another allocation debugging library called libgmalloc which seems to be more aggressive than LibMallocDebug. This library can be enabled through Xcode with the menu item “Enable Guard Malloc” in the menu “Run”. You can get more information about this library in the man page (man libgmalloc).