Testing if an arbitrary pointer is a valid Objective-C object

Posted: November 24th, 2016 | Author: | Filed under: Debugging, iOS, macOS, Programming | Tags: , , , , , , , , , | No Comments »

Let’s say you pick a random pointer. Can we know if it points to a valid Objective-C object? Of course without crashing… Well there is no simple solution. In this post I give a solution for 64-bit architectures. The code provided has only been tested on macOS 10.12.1 and iOS 10.1.1 with the modern Objective-C runtime.

There is not much documentation available on this subject. There is one article written in 2010 by Matt Gallagher but the content is outdated and not working properly anymore. Most of the information in this post comes from:

Disclaimer

The content of this post – as well as the source code – relies on internal structures of the Objective-C runtime. There is no guarantee on the correctness and might possibly break with any macOS/iOS update. Don’t use this code in real applications!

In fact I started to write this post based on the objc4-680 sources (mac OS 10.11.6). But just before publishing, Apple released the sources for objc4-706 (macOS 10.12). As you can see in the image below, some internal structures I rely on have been changed:


Changes between objc4-680 and objc4-706

What is a pointer?

A pointer is just an integer referencing a location in memory. On iOS and macOS, there are however 2 kinds of pointers: regular pointers and tagged pointers. Let’s start by solving the problem for tagged pointers.

Tagged Pointers

Tagged pointers were introduced in iOS 7 and Mac OS X 10.7 for 64-bit architectures. A tagged pointer is a special pointer with data stored directly into the pointer instead of doing memory allocations. This has obvious performance advantages.

Tagged pointers are declared in objc-internal.h. In macOS 10.11 and earlier tagged pointers were simple:

  • 60 bits for the payload
  • 3 bits for the tag index
  • 1 bit to identify tagged pointer objects vs. ordinary objects

In macOS 10.12 the tagged pointer layout has been changed to also support 52 bits payload and more tag indexes:

// Tag indexes 0..<7 have a 60-bit payload.
// Tag index 7 is reserved.
// Tag indexes 8..<264 have a 52-bit payload.
// Tag index 264 is reserved.

The tag index tells you the class represented by the tagged pointer:

{
    OBJC_TAG_NSAtom            = 0, 
    OBJC_TAG_1                 = 1, 
    OBJC_TAG_NSString          = 2, 
    OBJC_TAG_NSNumber          = 3, 
    OBJC_TAG_NSIndexPath       = 4, 
    OBJC_TAG_NSManagedObjectID = 5, 
    OBJC_TAG_NSDate            = 6, 
    OBJC_TAG_RESERVED_7        = 7, 

    OBJC_TAG_First60BitPayload = 0, 
    OBJC_TAG_Last60BitPayload  = 6, 
    OBJC_TAG_First52BitPayload = 8, 
    OBJC_TAG_Last52BitPayload  = 263, 

    OBJC_TAG_RESERVED_264      = 264
};

Checking if a pointer is a tagged pointer is really simple using the functions declared in objc-internal.h:


static inline bool 
_objc_isTaggedPointer(const void *ptr) 
{
    return ((intptr_t)ptr & _OBJC_TAG_MASK) == _OBJC_TAG_MASK;
}

static inline objc_tag_index_t 
_objc_getTaggedPointerTag(const void *ptr) 
{
    // assert(_objc_isTaggedPointer(ptr));
    uintptr_t basicTag = ((uintptr_t)ptr >> _OBJC_TAG_INDEX_SHIFT) & _OBJC_TAG_INDEX_MASK;
    uintptr_t extTag =   ((uintptr_t)ptr >> _OBJC_TAG_EXT_INDEX_SHIFT) & _OBJC_TAG_EXT_INDEX_MASK;
    if (basicTag == _OBJC_TAG_INDEX_MASK) {
        return (objc_tag_index_t)(extTag + OBJC_TAG_First52BitPayload);
    } else {
        return (objc_tag_index_t)basicTag;
    }
}

Sadly these functions are static inline and not exported and we have no other choice but to copy the implementations in our source code.

On the other hand the function to get the registered class for the given tag is exported and can be used:


/**
 Returns the registered class for the given tag.
 Returns nil if the tag is valid but has no registered class.

 This function searches the exported function: _objc_getClassForTag(objc_tag_index_t tag)
 declared in https://opensource.apple.com/source/objc4/objc4-706/runtime/objc-internal.h
 */
static Class _objc_getClassForTag(objc_tag_index_t tag)
{
	static bool _objc_getClassForTag_searched = false;
	static Class (*_objc_getClassForTag_func)(objc_tag_index_t) = NULL;
	if(!_objc_getClassForTag_searched)
	{
		_objc_getClassForTag_func = (Class(*)(objc_tag_index_t))dlsym(RTLD_DEFAULT, "_objc_getClassForTag");
		_objc_getClassForTag_searched = true;
		if(_objc_getClassForTag_func == NULL)
		{
			fprintf(stderr, "*** Could not find _objc_getClassForTag()!\n");
		}
	}
	
	if(_objc_getClassForTag_func != NULL)
	{
		return _objc_getClassForTag_func(tag);
	}
	
	return NULL;
}


It is now simple to create a function that checks if a pointer is a tagged pointer and thus a valid Objective-C object:

/**
 Test if a pointer is a tagged pointer

 @param inPtr is the pointer to check
 @param outClass returns the registered class for the tagged pointer.
 @return true if the pointer is a tagged pointer.
 */
bool IsObjcTaggedPointer(const void *inPtr, Class *outClass)
{
	bool isTaggedPointer = _objc_isTaggedPointer(inPtr);
	if(outClass != NULL)
	{
		if(isTaggedPointer)
		{
			objc_tag_index_t tagIndex = _objc_getTaggedPointerTag(inPtr);
			*outClass = _objc_getClassForTag(tagIndex);
		}
		else
		{
			*outClass = NULL;
		}
	}
	
	return isTaggedPointer;
}

If you want to read more about tagged pointers you can read these 2 articles from Mike Ash:

Now that we handled tagged pointers, let’s look at ‘regular’ pointers.

Alignment

Valid pointers have to be aligned to the pointer size. Such a check is done in LLDB when trying to print a pointer in the python function ‘is_valid_pointer’ in objc_runtime.py:

    @staticmethod
    def is_valid_pointer(pointer, pointer_size, allow_tagged=0, allow_NULL=0):
        logger = lldb.formatters.Logger.Logger()
        if pointer is None:
            return 0
        if pointer == 0:
            return allow_NULL
        if allow_tagged and (pointer % 2) == 1:
            return 1
        return ((pointer % pointer_size) == 0)

The last check verifies that the pointer is aligned to the pointer size. We can implement the same check:

if (((uintptr_t)inPtr % sizeof(uintptr_t)) != 0)
{
  return false;
}

Bits used

The LLDB source code has another interesting function in objc_runtime.py:

    # Objective-C runtime has a rule that pointers in a class_t will only have bits 0 thru 46 set
    # so if any pointer has bits 47 thru 63 high we know that this is not a
    # valid isa
    @staticmethod
    def is_allowed_pointer(pointer):
        logger = lldb.formatters.Logger.Logger()
        if pointer is None:
            return 0
        return ((pointer & 0xFFFF800000000000) == 0)

Again we can easily implement the same check:

if(((uintptr_t)inPtr & 0xFFFF800000000000) != 0)
{
  return false;
}

Valid and readable memory

In order to be valid, a pointer should point to valid and readable memory. We can use vm_region_64() to ensure that the memory is readable and vm_read() to ensure that the memory is valid:


/**
 Test if the pointer points to readable and valid memory.

 @param inPtr is the pointer
 @return true if the pointer points to readable and valid memory.
 */
static bool IsValidReadableMemory(const void *inPtr)
{
	kern_return_t error = KERN_SUCCESS;
	
	// Check for read permissions
	bool hasReadPermissions = false;
	
	vm_size_t vmsize;
	vm_address_t address = (vm_address_t)inPtr;
	vm_region_basic_info_data_t info;
	mach_msg_type_number_t info_count = VM_REGION_BASIC_INFO_COUNT_64;

	memory_object_name_t object;

	error = vm_region_64(mach_task_self(), &address, &vmsize, VM_REGION_BASIC_INFO, (vm_region_info_t)&info, &info_count, &object);
	if(error != KERN_SUCCESS)
	{
		// vm_region/vm_region_64 returned an error
		hasReadPermissions = false;
	}
	else
	{
		hasReadPermissions = (info.protection & VM_PROT_READ);
	}
	
	if(!hasReadPermissions)
	{
		return false;
	}
	
	// Read the memory
	vm_offset_t readMem = 0;
	mach_msg_type_number_t size = 0;
	error = vm_read(mach_task_self(), (vm_address_t)inPtr, sizeof(uintptr_t), &readMem, &size);
	if(error != KERN_SUCCESS)
	{
		// vm_read returned an error
        return false;
    }
	
    return true;
}

Validating the isa pointer and getting the Class pointer

Now that we know the address points to valid and readable memory, we can extract the possible isa pointer and get the class pointer. This is documented by Greg Parker in [objc explain]: Non-pointer isa:

If you are writing a debugger-like tool, the Objective-C runtime exports some variables to help decode isa fields. objc_debug_isa_class_mask describes which bits are the class pointer: (isa & class_mask) == class pointer. objc_debug_isa_magic_mask and objc_debug_isa_magic_value describe some bits that help distinguish valid isa fields from other invalid values: (isa & magic_mask) == magic_value for isa fields that are not raw class pointers. These variables may change in the future so do not use them in application code.

Here is the implementation to validate the isa pointer and extract the Class pointer:


uintptr_t isa = (*(uintptr_t *)inPtr);
Class ptrClass = NULL;

if ((isa & ~ISA_MASK) == 0)
{
	ptrClass = (Class)isa;
}
else
{
	if ((isa & ISA_MAGIC_MASK) == ISA_MAGIC_VALUE)
	{
		ptrClass = (Class)(isa & ISA_MASK);
	}
	else
	{
		ptrClass = (Class)isa;
	}
}

if(ptrClass == NULL)
{
	return false;
}


Verifying if the class exists

Now that we have the class, we can check if the Objective-C runtime knows it:

bool isKnownClass = false;

unsigned int numClasses = 0;
Class *classesList = objc_copyClassList(&numClasses);
for (int i = 0; i < numClasses; i++)
{
	if (classesList[i] == ptrClass)
	{
		isKnownClass = true;
		break;
	}
}
free(classesList);

if(!isKnownClass)
{
	return false;
}

Filtering out some false positives

A good trick from Greg Parker consists of filtering out some false positives by checking if the pointer malloc’ed size is greater than the class instance size:

size_t pointerSize = malloc_size(inPtr);
if(pointerSize > 0 && pointerSize < class_getInstanceSize(ptrClass))
{
	return false;
}

Summing it up

We now have all the elements to build a function that returns a boolean indicating if a pointer is an Objective-C object:


/**
 Test if a pointer is an Objective-C object

 @param inPtr is the pointer to check
 @return true if the pointer is an Objective-C object
 */
bool IsObjcObject(const void *inPtr)
{
	//
	// NULL pointer is not an Objective-C object
	//
	if(inPtr == NULL)
	{
		return false;
	}
	
	//
	// Check for tagged pointers
	//
	if(IsObjcTaggedPointer(inPtr, NULL))
	{
		return true;
	}
	
	//
	// Check if the pointer is aligned
	//
	if (((uintptr_t)inPtr % sizeof(uintptr_t)) != 0)
	{
		return false;
	}
	
	//
	// From LLDB:
	// Objective-C runtime has a rule that pointers in a class_t will only have bits 0 thru 46 set
    // so if any pointer has bits 47 thru 63 high we know that this is not a valid isa
	// See http://llvm.org/svn/llvm-project/lldb/trunk/examples/summaries/cocoa/objc_runtime.py
	//
	if(((uintptr_t)inPtr & 0xFFFF800000000000) != 0)
	{
		return false;
	}
	
	//
	// Check if the memory is valid and readable
	//
	if(!IsValidReadableMemory(inPtr))
	{
		return false;
	}
	
	//
	// Get the Class from the pointer
	// From http://www.sealiesoftware.com/blog/archive/2013/09/24/objc_explain_Non-pointer_isa.html :
	// If you are writing a debugger-like tool, the Objective-C runtime exports some variables
	// to help decode isa fields. objc_debug_isa_class_mask describes which bits are the class pointer:
	// (isa & class_mask) == class pointer.
	// objc_debug_isa_magic_mask and objc_debug_isa_magic_value describe some bits that help
	// distinguish valid isa fields from other invalid values:
	// (isa & magic_mask) == magic_value for isa fields that are not raw class pointers.
	// These variables may change in the future so do not use them in application code.
	//
	
	uintptr_t isa = (*(uintptr_t *)inPtr);
	Class ptrClass = NULL;
	
	if ((isa & ~ISA_MASK) == 0)
	{
		ptrClass = (Class)isa;
	}
	else
	{
		if ((isa & ISA_MAGIC_MASK) == ISA_MAGIC_VALUE)
		{
			ptrClass = (Class)(isa & ISA_MASK);
		}
		else
		{
			ptrClass = (Class)isa;
		}
	}
	
	if(ptrClass == NULL)
	{
		return false;
	}
	
	//
	// Verifies that the found Class is a known class.
	//
    bool isKnownClass = false;
	
	unsigned int numClasses = 0;
	Class *classesList = objc_copyClassList(&numClasses);
	for (int i = 0; i < numClasses; i++)
    {
		if (classesList[i] == ptrClass)
        {
            isKnownClass = true;
            break;
        }
    }
    free(classesList);
	
	if(!isKnownClass)
	{
		return false;
	}
	
	
	//
	// From Greg Parker
	// https://twitter.com/gparker/status/801894068502433792
	// You can filter out some false positives by checking malloc_size(obj) >= class_getInstanceSize(cls).
	//
	size_t pointerSize = malloc_size(inPtr);
	if(pointerSize > 0 && pointerSize < class_getInstanceSize(ptrClass))
	{
		return false;
	}
	
    return true;
}

Testing

To test this function, I built a simple iOS application that checks various pointers.
Here is the output when running on iOS 10.1.1 (64-bit):


Tests

Downloads

References

Updates

24.11.2016: 2 changes based on feedback from Greg Parker:

  • Filtering out some false positives
  • Use objc_copyClassList() instead of objc_getClassList()

Apple’s use of Swift in iOS 10.1 and macOS 10.12

Posted: November 1st, 2016 | Author: | Filed under: Analysis, iOS, macOS, Programming, Swift | 3 Comments »

Swift has been announced at the WWDC 2014, more than 2 years ago. Most of the sample code projects from Apple are now written in Swift. But does Apple use Swift in iOS 10.1 and macOS 10.12.1?

How to detect if a binary is using Swift?

A naïve approach would be to check if an app contains the Swift libraries in its Frameworks folder: libswiftCore.dylib, libswiftFoundation.dylib, …

Here is the content of the Frameworks folder of the MRT.app on macOS 10.12.1
/System/Library/CoreServices/MRT.app/Contents/Frameworks/ :


MRT.app

However this is not a good approach since iOS and macOS contain a private copy of the Swift libraries in /System/Library/PrivateFrameworks/Swift/ . Several apps in iOS and macOS link directly to these system libraries.

Here is the content of the Frameworks folder of the PIPAgent.app on macOS 10.12.1
/System/Library/CoreServices/PIPAgent.app/Contents/Frameworks/ :


PIPAgent.app

A much better approach is to check whether a binary links to a Swift library. This can easily be done with the command line tool ‘otool’ using the -L option:

-L Display the names and version numbers of the shared libraries that the object file uses, as well as the shared library ID if the file is a shared library.

When running this command on the PIPAgent application:

otool -L /System/Library/CoreServices/PIPAgent.app/Contents/MacOS/PIPAgent | grep swift

you would get the following output:

/System/Library/PrivateFrameworks/Swift/libswiftAppKit.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftCore.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftCoreData.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftCoreImage.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftFoundation.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 800.8.18)
/System/Library/PrivateFrameworks/Swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 800.8.18)

Building a script

Using the otool command line tool, it is easy to write a bash function that tells if a file is a binary linked to the Swift libraries:


#------------------------------------------------------------------------
# Function to check if a file (passed as argument $1) is using Swift
# It returns the number of occurrences of the string 'swift'
# from the output of otool
#------------------------------------------------------------------------
isFileUsingSwift ()
{
	otool -L $1 2>/dev/null | grep -o swift | wc -l
}

The processFile bash function takes a file as parameter and will print its path if it’s a binary linked to the Swift libraries:


#------------------------------------------------------------------------
# Function to process a file (passed as argument $1).
# It calls the function isFileUsingSwift() to determine
# if this is a binary using Swift and in this case
# print the path of this file.
#------------------------------------------------------------------------
processFile ()
{
	isFileUsingSwift=$( isFileUsingSwift $1 )
 	if [ ${isFileUsingSwift} != 0 ]
 	then
 		# We found a binary using Swift
	 	echo "   $1"
	fi
}


Looping through all the files of a folder is now a single line:

find ${PATH_TO_CHECK} -type f -exec bash -c 'processFile "$0"' {} \;

Final script

Below is the complete bash script that loops through all the files of a folder and print the paths of all the binaries found that use Swift.
Note: You can download the complete script here.


#!/bin/bash

#---------------------------------------------------------------------
# Bash script that loops through all the files of a folder and
# print the paths of all the binaries found that use Swift
# Created by Alexandre Colucci on 01.11.2016
# http://blog.timac.org/?p=1398
#---------------------------------------------------------------------


#---------------------------------------------------------------------
# Force expand a wildcard pattern into the list of matching pathnames
#---------------------------------------------------------------------
shopt -s nullglob

#---------------------------------------------------------------------
# Function to print the usage
#---------------------------------------------------------------------
printUsage ()
{
	echo "Usage: detectSwift.sh PATH"
	echo "PATH: Folder to search for binaries using Swift"
	echo ""
	echo "Examples:"
	echo "  detectSwift.sh /System/Library"
	echo "  detectSwift.sh /System"
	echo "  detectSwift.sh /"
	echo ""
	echo "Note: run as root in order to avoid permission issues."
	echo ""
}

#---------------------------------------------------------------------
# Function to check if a file (passed as argument $1) is using Swift
# It returns the number of occurrences of the string 'swift'
# from the output of otool
#---------------------------------------------------------------------
isFileUsingSwift ()
{
	otool -L $1 2>/dev/null | grep -o swift | wc -l
}

#---------------------------------------------------------------------
# Function to process a file (passed as argument $1).
# It calls the function isFileUsingSwift() to determine
# if this is a binary using Swift and in this case
# print the path of this file.
#---------------------------------------------------------------------
processFile ()
{
	isFileUsingSwift=$( isFileUsingSwift $1 )
 	if [ ${isFileUsingSwift} != 0 ]
 	then
 		# We found a binary using Swift
	 	echo "   $1"
	fi
}

#---------------------------------------------------------------------
# Check if the script was called with the expected usage
#---------------------------------------------------------------------
PARAMETER_NUMBER=$#
PARAMETER_REQUIRED=1
if [ $PARAMETER_NUMBER != $PARAMETER_REQUIRED ];
then
	printUsage
	exit 1
fi


#---------------------------------------------------------------------
# Get the folder path
#---------------------------------------------------------------------
PATH_TO_CHECK=$1

echo ""
echo "Start time:"
date
echo ""
echo "Apps using Swift in ${PATH_TO_CHECK}"


#---------------------------------------------------------------------
# Export the functions so that the subshell inherits them
#---------------------------------------------------------------------
export -f isFileUsingSwift
export -f processFile

#---------------------------------------------------------------------
# Find all the regular files in all subdirectories
# and call for each the function processFile()
#---------------------------------------------------------------------

find ${PATH_TO_CHECK} -type f -exec bash -c 'processFile "$0"' {} \;


#---------------------------------------------------------------------
# Finalizing
#---------------------------------------------------------------------
echo ""
echo "Completed at:"
date
echo ""

Running the script

The script is really slow: for each regular file it will create a subshell, call otool, grep and wc.
Running this script on the iOS 10.1 filesystem takes around 30 minutes.

For macOS 10.12.1, running the script on / takes dozen of hours. I recommend to only run this script on /System, /Applications and /usr. Processing these 3 folders in parallel will take around 2 hours.

Apple’s use of Swift in iOS 10.1

Running the script on iOS 10.1 (14B72c) of an iPhone 7 Plus will give you this list of binaries:

/Applications/Calculator.app/Calculator
/Applications/Music.app/Music
/Applications/Music.app/PlugIns/MusicMessagesApp.appex/MusicMessagesApp
/Applications/Music.app/PlugIns/RecentlyPlayedTodayExtension.appex/RecentlyPlayedTodayExtension
/System/Library/PrivateFrameworks/UpNextWidget.framework/PlugIns/UpNext.appex/UpNext

You will get these additional binaries from the dyld shared cache:

/System/Library/PrivateFrameworks/CoreKnowledge.framework/CoreKnowledge
/System/Library/PrivateFrameworks/Swift/libswiftAssetsLibrary.dylib
/System/Library/PrivateFrameworks/Swift/libswiftAVFoundation.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCloudKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftContacts.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCore.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreAudio.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreBluetooth.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreData.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreGraphics.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreImage.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreLocation.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreMedia.dylib
/System/Library/PrivateFrameworks/Swift/libswiftDarwin.dylib
/System/Library/PrivateFrameworks/Swift/libswiftDispatch.dylib
/System/Library/PrivateFrameworks/Swift/libswiftEventKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftFoundation.dylib
/System/Library/PrivateFrameworks/Swift/libswiftGameKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftGameplayKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftGLKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftHomeKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftLocalAuthentication.dylib
/System/Library/PrivateFrameworks/Swift/libswiftMultipeerConnectivity.dylib
/System/Library/PrivateFrameworks/Swift/libswiftObjectiveC.dylib
/System/Library/PrivateFrameworks/Swift/libswiftPassKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftRemoteMirror.dylib
/System/Library/PrivateFrameworks/Swift/libswiftSceneKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftsimd.dylib
/System/Library/PrivateFrameworks/Swift/libswiftSpriteKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftSwiftOnoneSupport.dylib
/System/Library/PrivateFrameworks/Swift/libswiftUIKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftWatchConnectivity.dylib
/System/Library/PrivateFrameworks/Swift/libswiftWatchKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftWebKit.dylib
/System/Library/PrivateFrameworks/UpNextWidget.framework/UpNextWidget

Note that you will get a similar output for an iPad except that the Calculator.app is not available.

Apple’s use of Swift in macOS 10.12.1

Running the script on macOS 10.12.1 will give you this list of binaries:

/Applications/Utilities/Console.app/Contents/MacOS/Console
/usr/bin/swift
/usr/bin/swiftc
/usr/sbin/usernoted
/System/Library/CoreServices/Dock.app/Contents/MacOS/Dock
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftAppKit.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftCore.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftCoreData.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftCoreGraphics.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftCoreImage.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftDarwin.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftDispatch.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftFoundation.dylib
/System/Library/CoreServices/MRT.app/Contents/Frameworks/libswiftObjectiveC.dylib
/System/Library/CoreServices/MRT.app/Contents/MacOS/MRT
/System/Library/CoreServices/NotificationCenter.app/Contents/MacOS/NotificationCenter
/System/Library/CoreServices/OSDUIHelper.app/Contents/MacOS/OSDUIHelper
/System/Library/CoreServices/PIPAgent.app/Contents/MacOS/PIPAgent
/System/Library/PrivateFrameworks/Swift/libswiftAppKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftAVFoundation.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCloudKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftContacts.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCore.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreAudio.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreBluetooth.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreData.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreGraphics.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreImage.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreLocation.dylib
/System/Library/PrivateFrameworks/Swift/libswiftCoreMedia.dylib
/System/Library/PrivateFrameworks/Swift/libswiftDarwin.dylib
/System/Library/PrivateFrameworks/Swift/libswiftDispatch.dylib
/System/Library/PrivateFrameworks/Swift/libswiftEventKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftFoundation.dylib
/System/Library/PrivateFrameworks/Swift/libswiftGameKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftGameplayKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftGLKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftIOKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftLocalAuthentication.dylib
/System/Library/PrivateFrameworks/Swift/libswiftMultipeerConnectivity.dylib
/System/Library/PrivateFrameworks/Swift/libswiftObjectiveC.dylib
/System/Library/PrivateFrameworks/Swift/libswiftOpenCL.dylib
/System/Library/PrivateFrameworks/Swift/libswiftRemoteMirror.dylib
/System/Library/PrivateFrameworks/Swift/libswiftSceneKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftsimd.dylib
/System/Library/PrivateFrameworks/Swift/libswiftSpriteKit.dylib
/System/Library/PrivateFrameworks/Swift/libswiftSwiftOnoneSupport.dylib
/System/Library/PrivateFrameworks/Swift/libswiftWebKit.dylib

Note that you will get a lot of matches in Xcode 8.1. If you exclude the various toolchains and platform SDKs, you will get:

/Applications/Xcode.app/Contents/Frameworks/IDEDocumentation.framework/Versions/A/IDEDocumentation
/Applications/Xcode.app/Contents/Frameworks/libswiftAppKit.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftAVFoundation.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftCore.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftCoreAudio.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftCoreData.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftCoreGraphics.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftCoreImage.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftCoreMedia.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftDarwin.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftDispatch.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftFoundation.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftIOKit.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftObjectiveC.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftQuartzCore.dylib
/Applications/Xcode.app/Contents/Frameworks/libswiftXPC.dylib
/Applications/Xcode.app/Contents/PlugIns/IDEDocViewer.ideplugin/Contents/MacOS/IDEDocViewer
/Applications/Xcode.app/Contents/PlugIns/IDELanguageSupportUI.ideplugin/Contents/MacOS/IDELanguageSupportUI
/Applications/Xcode.app/Contents/PlugIns/IDEQuickHelp.ideplugin/Contents/MacOS/IDEQuickHelp
/Applications/Xcode.app/Contents/PlugIns/XcodeBuiltInExtensions.appex/Contents/MacOS/XcodeBuiltInExtensions
/Applications/Xcode.app/Contents/SharedFrameworks/DNTDocumentationModel.framework/Versions/A/DNTDocumentationModel
/Applications/Xcode.app/Contents/SharedFrameworks/DNTDocumentationSupport.framework/Versions/A/DNTDocumentationSupport
/Applications/Xcode.app/Contents/SharedFrameworks/DNTSourceKitSupport.framework/Versions/A/DNTSourceKitSupport
/Applications/Xcode.app/Contents/SharedFrameworks/DNTTransformer.framework/Versions/A/DNTTransformer
/Applications/Xcode.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/DVTDocumentation
/Applications/Xcode.app/Contents/SharedFrameworks/DVTMarkup.framework/Versions/A/DVTMarkup
/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/repl_swift
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCore.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftCoreGraphics.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftDarwin.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftDispatch.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftFoundation.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftIOKit.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/Frameworks/libswiftObjectiveC.dylib
/Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework/Versions/A/SourceKit

Conclusion

Apple’s use of Swift in iOS 10.1 and macOS 10.12.1 is extremely limited.
On iOS 10.1 there are only 2 apps and 2 private frameworks using Swift:

  • Calculator.app (iPhone only)
  • Music.app
  • UpNextWidget.framework
  • CoreKnowledge.framework

On macOS 10.12.1 the list of apps using Swift is limited to:

  • Console
  • swift
  • swiftc
  • usernoted
  • Dock.app
  • MRT.app
  • NotificationCenter.app
  • OSDUIHelper.app
  • PIPAgent.app
  • (Xcode.app)

Analysis of the Facebook.app for iOS

Posted: October 18th, 2016 | Author: | Filed under: Analysis, Debugging, iOS, Programming | Tags: , , , , | 1 Comment »

Did you ever wonder why the Facebook.app for iOS is such a big download? This post tries to give some answers. The version 66.0 (released on 7 October 2016) was analyzed on an iPad Air 2 (64-bit).

Here is what you see when downloading Facebook on an iPad Air 2:

Download Info

App content

A scan of the content of the Facebook app using GrandPerspective gives already a good overview:


GrandPerspective

As you can see more than 100 MB are taken by the binary itself. The images (Assets.car), the localizations and other resources are only responsible for a small part of the size.

One note: The Facebook app seems to use the open source JSC framework. Unless there is a technical reason to use it, Facebook could remove this framework and use instead the JavaScriptCore framework built in iOS. JavaScriptCore is available on iOS 7 and later and Facebook targets iOS 8 and later.

 

Slicing

One important point to note is that the Facebook binary I downloaded only contains 64-bit code. There is no 32-bit code. Facebook uses the Slicing mechanism which is described by Apple here:

Slicing is the process of creating and delivering variants of the app bundle for different target devices. A variant contains only the executable architecture and resources that are needed for the target device.

 

Entropy

Let’s look at the Facebook binary. The entropy graph generated by Hopper gives a good idea of the content:


Entropy

The red part (with biggest entropy) is actual code while the green and orange parts are other sections (strings, constants, …). Just by looking at the entropy graph, we know that around 50% of the binary is code.

 

Sections

By looking at the sections of the binary we indeed see that the (__TEXT, __text) section is more than 50 MB. Here is the list of sections sorted by size:

(__TEXT, __text): 56.60 MB
(__DATA, __objc_const): 16.57 MB
(__RODATA, __cstring): 5.02 MB
(__RODATA, __objc_methname): 4.55 MB
(__DATA, __const): 4.46 MB
(__RODATA, __gcc_except_tab): 3.87 MB
(__RODATA, __objc_methtype): 2.66 MB
(__DATA, __objc_data): 2.41 MB
(__DATA, __cfstring): 2.20 MB
(__DATA, __data): 1.66 MB
(__TEXT, __unwind_info): 1.58 MB
(__RODATA, __objc_classname): 1.42 MB
(__RODATA, __const): 1.40 MB
(__DATA, __bss): 0.89 MB
(__DATA, __objc_selrefs): 0.80 MB
(__DATA, __common): 0.54 MB
(__DATA, __objc_ivar): 0.38 MB
(__DATA, __objc_classlist): 0.24 MB
(__DATA, __objc_classrefs): 0.18 MB
(__DATA, __objc_superrefs): 0.13 MB
(__TEXT, __const): 0.07 MB
(__DATA, __objc_protolist): 0.06 MB
(__TEXT, __stubs): 0.02 MB
(__TEXT, __stub_helper): 0.02 MB
(__DATA, __la_symbol_ptr): 0.01 MB
(__DATA, __got): 0.01 MB
(__TEXT, __ustring): 0.01 MB
(__DATA, fbsessiongks): 0.01 MB
(__TEXT, __eh_frame): 0.01 MB
(__DATA, __mod_init_func): 0.00 MB
(__DATA, __objc_protorefs): 0.00 MB
(__DATA, __objc_catlist): 0.00 MB
(__DATA, __objc_nlclslist): 0.00 MB
(__DATA, FBInjectable): 0.00 MB
(__DATA, __mod_term_func): 0.00 MB
(__DATA, __objc_imageinfo): 0.00 MB

 

__RODATA segment

If you read carefully the list of sections, you did notice that the Facebook.app has a __RODATA segment containing sections generally found inside the __TEXT segment. The reason for this oddity is to work around an App Review limitation. As you can read on the App Review page:

Each Mach-O executable file (for example, app_name.app/app_name) must not exceed these limits:

  • For apps whose MinimumOSVersion is less than 7.0: maximum of 80 MB for the total of all __TEXT sections in the binary.
  • For apps whose MinimumOSVersion is 7.x through 8.x: maximum of 60 MB per slice for the __TEXT section of each architecture slice in the binary.
  • For apps whose MinimumOSVersion is 9.0 or greater: maximum of 400 MB for the size of the Mach-O binary file.

 

Facebook targets iOS 8.0 and later and as such its __TEXT segment should not exceed 60 MB. But if you sum up the __TEXT and __RODATA segments, you end up with more than 77 MB:

Total size of the __TEXT sections: 58.3 MB
Total size of the __RODATA sections: 18.9 MB

Facebook avoids this limitation by moving some if the __TEXT sections into the read only __RODATA segment. Implementing this trick is really simple: you just need to add a linker flag to rename the chosen sections. And it appears you need absolutely nothing at runtime: the renamed sections will be found automatically. This linker flag is described in the ld man page:

-rename_section orgSegment orgSection newSegment newSection
Renames section orgSegment/orgSection to newSegment/newSection.

You could use it to rename the (__TEXT, __cstring) section to (__RODATA, __cstring) by simply adding this line into the Other Linker Flags (OTHER_LDFLAGS):

-Wl,-rename_section,__TEXT,__cstring,__RODATA,__cstring

You can download a sample macOS Xcode project here. If you load the original binary (without the section rename) using MachOView, you will see the (__TEXT, __cstring) section:


Standard Section

After adding the linker flag and recompiling, a (__RODATA, __cstring) section will be visible and the binary will run as expected:

Renamed Section

Note that this workaround is only needed as long as Facebook supports iOS 8. When targeting iOS 9 or later, the limitation is 400 MB which should be enough without splitting the __TEXT segment.

 

Third party libraries

So what does the code contain? Sadly I could not find a license/readme file containing the list of third party libraries. Also the Info view in Settings.app > Facebook > Settings > Infos is desperately empty – this seems like a Facebook.app bug.

By looking at the class names, I could build a list of third party libraries used. This list is most likely incomplete but hopefully correct:

  • AsyncDisplayKit: http://asyncdisplaykit.org
    keeps even the most complex user interfaces smooth and responsive
  • Bolts-ObjC: https://github.com/BoltsFramework/Bolts-ObjC
    Bolts is a collection of low-level libraries designed to make developing mobile apps easier
  • CocoaAsyncSocket: https://github.com/robbiehanson/CocoaAsyncSocket
    Asynchronous socket networking library for Mac and iOS
  • CocoaLumberjack: https://github.com/CocoaLumberjack/CocoaLumberjack
    A fast & simple, yet powerful & flexible logging framework for Mac and iOS
  • ComponentKit: http://componentkit.org
    ComponentKit takes a functional, declarative approach to building UI
  • EGODatabase: https://github.com/enormego/egodatabase
    EGODatabase is a thread-safe Objective-C SQLite wrapper with full support for asynchronous SQLite calls as well as built in NSOperationQueue support
  • FLAnimatedImage: https://github.com/Flipboard/FLAnimatedImage
    Performant animated GIF engine for iOS
  • FXBlurView: https://github.com/nicklockwood/FXBlurView
    UIView subclass that replicates the iOS 7 realtime background blur effect, but works on iOS 5 and above
  • Google Toolbox for Mac: https://github.com/google/google-toolbox-for-mac
    Google Toolbox for Mac
  • JSONKit: https://github.com/johnezang/JSONKit
    Objective-C JSON
  • libphonenumber: https://github.com/iziz/libPhoneNumber-iOS
    iOS port from libphonenumber
  • OAuth2Client: https://github.com/nxtbgthng/OAuth2Client
    Client library for OAuth2
  • OCMock: http://ocmock.org
    Mock objects for Objective-C
  • pop: https://github.com/facebook/pop
    An extensible iOS and OS X animation library, useful for physics-based interactions.
  • nighthawk-webrtc: https://github.com/ceaglest/nighthawk-webrtc
    Custom builds of WebRTC for iOS
  • React Native: https://facebook.github.io/react-native/
    Build Native Mobile Apps using JavaScript and React
  • ZipArchive: https://github.com/ZipArchive/ZipArchive
    ZipArchive is a simple utility class for zipping and unzipping files on iOS and Mac
  • ZipZap: https://github.com/pixelglow/ZipZap
    zip file I/O library for Mac OS X and iOS

 

Objective-C awards

The awards for the best Objective-C classes is attributed to:

NTNativeTemplateHackyWorkaroundBecauseNSMapTableIsBuggyContainer

The award for the longest Objective-C method is attributed to:

- (void)[FBComposerCompositionStateMutation matchBootstrap:
		activityAttachmentUpdatedMutation:
		activitySharingComposition:
		albumDescription:
		albumTitle:
		appProvidedHashtag:
		audienceMutation:
		audienceTouchedByUser:
		author:
		clearContent:
		containerPublishTargetForMediaCollection:
		composerCancelAnalytics:
		defaultAudienceMutation:
		destinationMutation:
		feedOnlyToggleState:
		hashtags:
		inspirationMutation:
		isImplicitLocationExplicitlyBlockedByUser:
		lifeEventDateMutation:
		lifeEventRelationshipUpdated:
		lifeEventTitleOrIconUpdated:
		linkShareMutation:
		shareOriginalPostMutation:
		removeMediaAttachmentWithAssetID:
		replaceAllExistingMediaAttachments:
		addSingleMediaAttachment:
		replaceSingleExistingMediaAttachmentIfExisting:
		mediaAttachmentCaptionMutation:
		mediaAttachmentMentionsMutation:
		mediaReorderingMutation:
		mentions:
		moviesComposition:
		moviesLoadedTaggedUsers:
		photoAutoTaggingState:
		photoTaggingAssetMutation:
		placeMutation:
		placeSuggestions:
		placeTaggingState:
		platformLoadedActionMutation:
		platformLoadedAppAttribution:
		platformLoadedImageAttachments:
		platformLoadedOGMediaAttachmentsMutation:
		platformLoadedPlaceMutation:
		platformLoadedRef:
		platformLoadedRobotext:
		platformLoadedTaggedUsers:
		platformLoadedVideoAttachment:
		postContentTypeMutation:
		postPromptPayload:
		publishAsQuestionAndAnswerSession:
		togglePublishAsQuestionAndAnswerSession:
		preferredMarketplaceMutation:
		productItemCategoryGraphQLID:
		productItemPickupDeliveryInfo:
		productItemPriceMutation:
		productItemShouldPostToMarketplace:
		productItemSuggestedPlace:
		productItemSuggestedZipCode:
		productItemTitle:
		publishTarget:
		remoteImageAttachments:
		removeSponsor:
		removeStorylineAttachment:
		removeStickerAttachment:
		replaceStorylineAttachmentWithMediaAttachments:
		selectedStarRating:
		shouldCreateProfileBadge:
		statusTextMutation:
		stickers:
		taggableActivityComposition:
		targeting:
		withTagsMutation:
		poll:
		sponsor:
		videoAttachmentMutation:
		videoTaggingAssetMutation:
		videoTaggingAnalytics:
		boostedComponentData:
		albumTracker:]

 

Conclusion

The Facebook app is a complex app and not just a simple web view. It contains a lot of third party libraries and even splits its __TEXT segment in order to not reach the 60 MB limit imposed by Apple.


Accessing the Temperature Unit setting in iOS 10

Posted: September 26th, 2016 | Author: | Filed under: iOS, Programming | Tags: , , , , , , , | No Comments »

In iOS 10 Apple added a new dedicated setting for Temperature Unit in the Settings.app under General > Language & Region > Temperature Unit . It lets you switch your preferred unit between Fahrenheit and Celsius:

iOS 10 settings

 

Sadly Apple did not provide a public API for third party apps. Here is how you can access this preference in your app:

1- You first need to expose the NSLocaleTemperatureUnit NSLocaleKey:

FOUNDATION_EXPORT NSLocaleKey const NSLocaleTemperatureUnit;

2- You can now print the temperature unit. The value is one of these 2 strings: ‘Celsius’ or ‘Fahrenheit’.

NSLog(@"NSLocaleTemperatureUnit value: %@", [[NSLocale currentLocale] objectForKey:NSLocaleTemperatureUnit]);

Dump decrypted mach-o apps

Posted: August 4th, 2016 | Author: | Filed under: code injection, DYLD_INSERT_LIBRARIES, macOS, Programming | Tags: , , , , , , , , | 2 Comments »

 
In a previous post “CryptedHelloWorld: App with encrypted mach-o sections”, I created a simple macOS app ‘CryptedHelloWorld’ with its (__TEXT, __text) section encrypted. The section is decrypted by a constructor function.

 
This post explains how to dump the decrypted app. A common way is to attach the app with a debugger (GDB, LLDB) and manually dump the decrypted memory to disk.

However I will use a different solution by using 2 techniques already presented in previous posts: a destructor function and code injection.

The targeted app is the precompiled CryptedHelloWorld command line tool that can be downloaded here.

This command line tool has its (__TEXT, __text) section encrypted. Once its main() function is called, we know that the section is decrypted. Thus we can create a destructor function – which is called just before the app is quit – to dump the decrypted memory to disk. This destructor function will be injected into the app using the DYLD_INSERT_LIBRARIES environment variable.

The destructor function needs to read the executable from disk, dump the decrypted (__TEXT, __text) section and replace the encrypted bytes by the decrypted bytes.

 
 
Code injection

I already described how to inject code using DYLD_INSERT_LIBRARIES in this post. We will use the exact same technique by building a dynamic library like this:

gcc -o DumpBinary.dylib -dynamiclib DumpBinary.c

and then run it using the DYLD_INSERT_LIBRARIES environment variable:

DYLD_INSERT_LIBRARIES=./DumpBinary.dylib ./CryptedHelloWorld

 
 
Destructor function

Creating a destructor function has been described in this post. Such a function will be called just before the app quits:

void __attribute__((destructor)) DumpBinaryDestructor()
{
     // Executed just before the app quits
}

 
 
Finding the targeted app mach-o header

The first problem to solve is to find the mach-o header of the targeted app. This is easily done using the dyld function _dyld_get_image_header and searching for the first image of type MH_EXECUTE:


//
// Find the main executable
//
const struct mach_header_64 *machHeader = NULL;

for(uint32_t imageIndex = 0 ; imageIndex < _dyld_image_count() ; imageIndex++)
{
	const struct mach_header_64 *mH = (const struct mach_header_64 *)_dyld_get_image_header(imageIndex);
	if (mH->filetype == MH_EXECUTE)
	{
		const char* imageName = _dyld_get_image_name(imageIndex);
		fprintf(stderr, "Found main executable '%s'\n", imageName);
		
		machHeader = mH;
		break;
	}
}

 
 
Finding the executable path on disk

The dynamic library will need to read the app binary from disk: it needs the executable path. This is done using the dyld function _NSGetExecutablePath which copies the path of the main executable into a buffer:


//
// Get the real executable path
//
char executablePath[PATH_MAX];

/*
_NSGetExecutablePath() copies the path of the main executable into the
 buffer buf.  The bufsize parameter should initially be the size of the
 buffer.  This function returns 0 if the path was successfully copied, and
 * bufsize is left unchanged.  It returns -1 if the buffer is not large
 enough, and * bufsize is set to the size required.  Note that
 _NSGetExecutablePath() will return "a path" to the executable not a "real
 path" to the executable.  That is, the path may be a symbolic link and
 not the real file. With deep directories the total bufsize needed could
 be more than MAXPATHLEN.
*/
uint32_t len = sizeof(executablePath);
if (_NSGetExecutablePath(executablePath, &len) != 0)
{
	fprintf(stderr, "Buffer is not large enough to copy the executable path\n");
	exit(1);
}

We then get the canonical path using realpath:



//
// Get the canonicalized absolute path
//
char *canonicalPath = realpath(executablePath, NULL);
if (canonicalPath != NULL)
{
	strlcpy(executablePath, canonicalPath, sizeof(executablePath));
	free(canonicalPath);
}

 
 
Reading from disk

Reading from disk is done using fopen/fread:


//
// Open the executable file for reading
//
FILE *sourceFile = fopen(executablePath, "r");
if (sourceFile == NULL)
{
	fprintf(stderr, "Error: Could not open executable path '%s'\n", executablePath);
	exit(1);
}

//
// Read the source file and store it into a buffer
//
fseek(sourceFile, 0, SEEK_END);
long fileLen = ftell(sourceFile);
fseek(sourceFile, 0, SEEK_SET);

uint8_t *fileBuffer = (uint8_t *)calloc(fileLen, 1);
if (fileBuffer == NULL)
{
	fprintf(stderr, "Error: Could not allocate buffer\n");
	exit(1);
}

if (fread(fileBuffer, 1, fileLen, sourceFile) != fileLen)
{
	fprintf(stderr, "Error: Could not read the file '%s'\n", executablePath);
	exit(1);
}

 
 
Finding the (__TEXT, __text) and (__DATA, __mod_init_func) sections

We already have the mach-o header. We need to loop through all segments and sections until we find the interesting sections:


//
// Loop through each section
//
size_t segmentOffset = sizeof(struct mach_header_64);

for (uint32_t i = 0; i < machHeader->ncmds; i++)
{
	struct load_command *loadCommand = (struct load_command *)((uint8_t *) machHeader + segmentOffset);
	
	if(loadCommand->cmd == LC_SEGMENT_64)
	{
		// Found a 64-bit segment
		struct segment_command_64 *segCommand = (struct segment_command_64 *) loadCommand;

		// For each section in the 64-bit segment
		void *sectionPtr = (void *)(segCommand + 1);
		for (uint32_t nsect = 0; nsect < segCommand->nsects; ++nsect)
		{
			struct section_64 *section = (struct section_64 *)sectionPtr;
			
			fprintf(stderr, "Found the section (%s, %s)\n", section->segname, section->sectname);
			
			if (strncmp(segCommand->segname, SEG_TEXT, 16) == 0)
			{
				if (strncmp(section->sectname, SECT_TEXT, 16) == 0)
				{
					// This is the (__TEXT, __text) section.

				}
			}
			else if (strncmp(segCommand->segname, SEG_DATA, 16) == 0)
			{
				if (strncmp(section->sectname, "__mod_init_func", 16) == 0)
				{
					// This is the (__DATA, __mod_init_func) section.

				}
			}
			
			sectionPtr += sizeof(struct section_64);
		}
	}
	
	segmentOffset += loadCommand->cmdsize;
}



 
 
Dumping the decrypted (__TEXT, __text) section

We just use a simple memcpy to replace in the buffer the encrypted bytes by the decrypted bytes:

fprintf(stderr, "\t Save the unencrypted (%s, %s) section to the buffer\n", section->segname, section->sectname);
memcpy(fileBuffer + section->offset, (uint8_t *) machHeader + section->offset, section->size);

 
 
Removing the constructor function

We now have a decrypted binary. However if we launch it, its constructor function will be called and corrupt the (__TEXT, __text) section. We need to prevent the constructor function to be executed. There are several solutions and I chose to zero out the (__DATA, __mod_init_func) section. I kept the segname and sectname info so that MachOView can nicely display it.


fprintf(stderr, "\t Zero out the (%s, %s) section\n", section->segname, section->sectname);

size_t sectionOffset = sectionPtr - (void *)machHeader;

// Size of char sectname[16] + char segname[16]
size_t namesSize = 2 * 16 * sizeof(char);

// Zero out the section_64 but keep the sectname and segname
bzero(fileBuffer + sectionOffset + namesSize, sizeof(struct section_64) - namesSize);

 
 
Writing the decrypted binary to disk

The last step consists of writing the decrypted app to disk. This is done with fwrite:


//
// Create the output file
//
char destinationPath[PATH_MAX];
strlcpy(destinationPath, executablePath, sizeof(destinationPath));
strlcat(destinationPath, "_Decrypted", sizeof(destinationPath));

FILE *destinationFile = fopen(destinationPath, "w");
if (destinationFile == NULL)
{
	fprintf(stderr, "Error: Could create the output file '%s'\n", destinationPath);
	exit(1);
}

//
// Save the data into the output file
//
if (fwrite(fileBuffer, 1, fileLen, destinationFile) != fileLen)
{
	fprintf(stderr, "Error: Could not write to the output file\n");
	exit(1);
}


 
 
Log output

To run the CryptedHelloWorld app and inject our code:

DYLD_INSERT_LIBRARIES=./DumpBinary.dylib ./CryptedHelloWorld

Here is the log output when running the CryptedHelloWorld:


*** Constructor called to decrypt sections
Found the section (__TEXT, __text)
Decrypting the (__TEXT, __text) section
Found the section (__TEXT, __stubs)
Found the section (__TEXT, __stub_helper)
Found the section (__TEXT, __timac)
Found the section (__TEXT, __cstring)
Found the section (__TEXT, __unwind_info)
Found the section (__DATA, __nl_symbol_ptr)
Found the section (__DATA, __got)
Found the section (__DATA, __la_symbol_ptr)
Found the section (__DATA, __mod_init_func)
Found the section (__DATA, __data)
Found the section (__DATA, __bss)
*** The sections should now be decrypted. main() will be called soon.


------------------
Hello, World!
------------------


*********************************
*** DumpBinaryDestructor CALLED
*********************************
Found main executable '/CryptedHelloWorld/DumpBinary/./CryptedHelloWorld'
Found absolute path: '/CryptedHelloWorld/DumpBinary/CryptedHelloWorld'
Found the section (__TEXT, __text)
	 Save the unencrypted (__TEXT, __text) section to the buffer
Found the section (__TEXT, __stubs)
Found the section (__TEXT, __stub_helper)
Found the section (__TEXT, __timac)
Found the section (__TEXT, __cstring)
Found the section (__TEXT, __unwind_info)
Found the section (__DATA, __nl_symbol_ptr)
Found the section (__DATA, __got)
Found the section (__DATA, __la_symbol_ptr)
Found the section (__DATA, __mod_init_func)
	 Zero out the (__DATA, __mod_init_func) section
Found the section (__DATA, __data)
Found the section (__DATA, __bss)
*********************************
*** Decryption completed
*********************************

 
 
Examining the decrypted app

Using MachOView, we see that the (__TEXT, __text) section is decrypted:


(__TEXT, __text)

 

We also see that the (__DATA, __mod_init_func) has been zeroed out:


(__DATA, __mod_init_func)

 
 
Limitations of the dynamic library

  • it only supports 64-bit intel Mach-O files. Adding 32-bit, ARM or fat Mach-O support is fairly simple and left to the reader.
  • it only dumps the (__TEXT, __text) section.
  • it zeroes out the (__DATA, __mod_init_func) section which would cause problems if there are multiple constructors.

 
 
Downloads

The dynamic library source code can be downloaded here.

The precompiled dynamic library can be downloaded here.


Mail.app plugin compatibility for macOS Sierra (10.12)

Posted: July 27th, 2016 | Author: | Filed under: macOS, Mail, Programming | Tags: , , , , , , , , , | No Comments »

 
Mail.app in macOS 10.11 and earlier used to check the plugins compatibility using the SupportedPluginCompatibilityUUIDs key in the plugin’s Info.plist. For example a Mail plugin would only be compatible with macOS 10.11.6 if its Info.plist contained the following:

<key>SupportedPluginCompatibilityUUIDs</key>
<array>
   <string>71562B89-0D90-4588-8E94-A75B701D6443</string>
</array>

  
Mail.app version 10.0 in macOS Sierra (10.12) now uses a different key to check the plugins compatibility. It now requires a key with the format Supported%ld.%ldPluginCompatibilityUUIDs where “%ld.%ld” is the operating system version like “10.12”.

If you want to make your Mail.app plugin compatible with macOS Sierra, you will need to add a key in your Info.plist like:

<key>Supported10.12PluginCompatibilityUUIDs</key>
<array>
    <string>36CCB8BB-2207-455E-89BC-B9D6E47ABB5B</string>
</array>

Note that the UUID “36CCB8BB-2207-455E-89BC-B9D6E47ABB5B” is for macOS Sierra 10.12 beta 3.

  
Also by default the Mail plugins support is disabled. In order to enable it you first need to run in the Terminal the following:

defaults write com.apple.mail EnableBundles -bool true

  
Related post: Disable swipe to delete in Mail.app on OS X 10.11


CryptedHelloWorld: App with encrypted mach-o sections

Posted: July 23rd, 2016 | Author: | Filed under: crypto, macOS, Programming | Tags: , , , , , , , | No Comments »

 
In a previous post ( constructor and destructor attributes ), I described the constructor attribute and mentioned software protection as a possible use case:

A constructor attribute could be used to implement a software protection. You could encrypt your executable with a custom encryption and use a constructor function to decrypt the binary just before it is loaded.

In this post I describe such a protection with an example.

 
 
Mach-O file format

Let’s start with a brief summary of the Mach-O file format. For more information you should definitively read the OS X ABI Mach-O File Format Reference.

A Mach-O file contains 3 regions:

  • header structure: describes the Mach-O file
  • load commands: describes the segments and their sections
  • actual segment data

 
Here is the figure from the OS X ABI Mach-O File Format Reference:


Mach-O file format basic structure

The goal is to encrypt the data of the __text section from the __TEXT segment. This is the section containing the code of the executable. The other sections and segments will be left untouched.

 
 
The target application: CryptedHelloWorld

The target application is called ‘CryptedHelloWorld’. Its (__TEXT, __text) section is encrypted, meaning that the main() function needs to be decrypted before running. When launched, a constructor function will decrypt the encrypted section and the decrypted main() will be called.

The application itself is a simple command line ‘Hello World’ written in C. Here is the source code of the main function:


int main(int argc, const char * argv[])
{
	printf("\n\n");
	printf("------------------\n");
	printf("Hello, World!\n");
	printf("------------------\n");
	printf("\n\n");
	
    return 0;
}

 
 
CryptoTool

When building the project with Xcode, the target dependency ‘CryptoTool’ is built. At the end of the compilation of the CryptedHelloWorld app, a Run Script phase is executed which runs ‘CryptoTool’ on the just compiled CryptedHelloWorld binary. ‘CryptoTool’:

  • reads the binary of the CryptedHelloWorld application from the disk
  • locates the (__TEXT, __text) section in the file
  • encrypts it using AES 128
  • replaces the bytes with the encrypted bytes in the binary on the disk.

 
The Run Script phase is straightforward:


Run Script phase

 
 
Location of the constructor function

When you launch the application, the constructor function will be triggered and will need to decrypt the (__TEXT, __text) section. Obviously this constructor function can’t be located in the (__TEXT, __text) section. I store it in a custom (__TEXT,__timac) section using the attribute __attribute__((section(“__TEXT,__timac”))):

void __attribute__((constructor)) __attribute__((section("__TEXT,__timac"))) decryptTextSection()

 
 
Self contained constructor function

We need to make sure that the constructor function doesn’t call any functions that are located in the (__TEXT, __text) section. To ensure that I made all the required functions inlined. For example:

static inline void __attribute__((always_inline)) EncryptDecryptMachoFile(struct mach_header_64 *machHeader, CCOperation operation)

The attribute __attribute__((always_inline)) ensures that the function is inlined even for debug builds when optimizations are turned off.

 
 
Locating the (__TEXT, __text) section

The constructor function uses the _dyld_get_image_header() dyld function to get the Mach-O header. It then loops though all the commands and all the sections of the LC_SEGMENT_64 segments to find the (__TEXT, __text) section:

static inline void __attribute__((always_inline)) EncryptDecryptMachoFile(struct mach_header_64 *machHeader, CCOperation operation)
{
	size_t segmentOffset = sizeof(struct mach_header_64);
	
	// For each load command of the mach-o file
	for (uint32_t i = 0; i < machHeader->ncmds; i++)
	{
		struct load_command *loadCommand = (struct load_command *)((uint8_t *) machHeader + segmentOffset);
		if(loadCommand->cmd == LC_SEGMENT_64)
		{
			// We found a 64-bit segment
			struct segment_command_64 *segCommand = (struct segment_command_64 *) loadCommand;
			
			// For each section in the 64-bit segment
			void *sectionPtr = (void *)(segCommand + 1);
			for (uint32_t nsect = 0; nsect < segCommand->nsects; ++nsect)
			{
				struct section_64 *section = (struct section_64 *)sectionPtr;
				fprintf(stderr, "Found the section (%s, %s)\n", section->segname, section->sectname);
				
				// Check if this is the __TEXT segment
				if (strncmp(segCommand->segname, SEG_TEXT, 16) == 0)
				{
					// Check if this is the __text section
					if (strncmp(section->sectname, SECT_TEXT, 16) == 0)
					{
						// This is the (__TEXT, __text) section.
						// We should encrypt/decrypt it in place.
						fprintf(stderr, "%s the (%s, %s) section\n", (operation == kCCEncrypt) ? "Encrypting" : "Decrypting", section->segname, section->sectname);
						EncryptDecryptBuffer((uint8_t *) machHeader + section->offset, (uint8_t *) machHeader + section->offset, section->size, operation);
						
					}
				}
				
				sectionPtr += sizeof(struct section_64);
			}
		}
		
		segmentOffset += loadCommand->cmdsize;
	}
}

 
 
Encryption

The section is encrypted using AES 128 by chunks of PAGE_SIZE bytes (4096 bytes). If there are less than PAGE_SIZE bytes to encrypt, chunks of 16 bytes are used. I use the CommonCrypto implementation of AES 128:

static inline void __attribute__((always_inline)) EncryptDecryptBytes(const void *srcBuffer, void *dstBuffer, uint64_t len, CCOperation operation)
{
	// Encrypt/decrypt the data from the source buffer using the AES key
	size_t outLength = 0;
	CCCryptorStatus result = CCCrypt(operation,
                   kCCAlgorithmAES128,
                   kCCOptionPKCS7Padding,
                   sAesKey,
                   kCCKeySizeAES128,
                   NULL,
                   srcBuffer,
                   len,
                   sEncryptionBuffer,
                   len + kCCBlockSizeAES128,
                   &outLength);
	
	if (result == kCCSuccess)
	{
		// Copy the encrypted/decrypted data into the destination buffer
		memcpy(dstBuffer, sEncryptionBuffer, len);
	}
	else
	{
		fprintf(stderr, "Error %d: Could not %s the data\n", result, (operation == kCCEncrypt) ? "encrypt" : "decrypt");
		exit(1);
	}
}

 
 
Virtual memory protections

The __TEXT segment is not writable by default. In order to decrypt the memory in place, the virtual memory protections need to be changed to allow writes. This is done using vm_protect:


static inline void __attribute__((always_inline)) ChangeVirtualMemoryProtections(vm_address_t addr, vm_size_t size)
{
	kern_return_t returnValue = vm_protect(mach_task_self(), addr, size, false, VM_PROT_ALL);
	if ( returnValue != KERN_SUCCESS )
	{
		fprintf(stderr, "Error %d: Fail to change virtual memory protections\n", returnValue);
		exit(1);
	}
}

 
 
Testing the compiled application

Here is the log output when launching the binary using the Terminal:

*** Constructor called to decrypt sections
Found the section (__TEXT, __text)
Decrypting the (__TEXT, __text) section
Found the section (__TEXT, __stubs)
Found the section (__TEXT, __stub_helper)
Found the section (__TEXT, __timac)
Found the section (__TEXT, __cstring)
Found the section (__TEXT, __unwind_info)
Found the section (__DATA, __nl_symbol_ptr)
Found the section (__DATA, __got)
Found the section (__DATA, __la_symbol_ptr)
Found the section (__DATA, __mod_init_func)
Found the section (__DATA, __data)
Found the section (__DATA, __bss)
*** The sections should now be decrypted. main() will be called soon.

——————
Hello, World!
——————

 
 
Examining the compiled application

 
Using MachOView, we see that the (__TEXT, __text) section is encrypted:


Section __text

 
This is confirmed with Hopper. The main() function doesn’t make any sense:


Hopper

 
Back to MachOView, we see that the (__TEXT, __timac) section contains unencrypted code:


Section __timac

 
 
Limitations of this proof of concept

As mentioned this example is a proof of concept and has several limitations:

  • it only supports 64-bit Mach-O files. Adding 32-bit and fat Mach-O support is fairly simple and left to the reader.
  • only the (__TEXT, __text) section is encrypted. It is possible to encrypt other sections or maybe even the whole __TEXT segment.
  • the target application is a really simple command line application written in C.
  • As mentioned earlier all the functions required in the constructor function have been made inlined. This makes it difficult to debug. If you want to debug this code I recommend to debug the CryptoTool app. It supports the parameters -decrypt and -encrypt. You should also remove the always_inline attribute otherwise breakpoints won’t fire as you would expect.

 
 
Should you use such code to protect your application?

I wouldn’t. Although this might prevent a user to look at the code, this won’t defeat an experienced attacker. Also note that this is a proof of concept.

 
 
Downloads

You can download the precompiled CryptedHelloWorld command line tool here.

The whole source code is available here.


constructor and destructor attributes

Posted: July 16th, 2016 | Author: | Filed under: code injection, Debugging, macOS, Programming | Tags: , , , , , , | 3 Comments »

GCC (and Clang) supports constructor and destructor attributes:

__attribute__((constructor))
__attribute__((destructor))

 
 
Description

A function marked with the __attribute__((constructor)) attribute will be called automatically before your main() function is called. Similarly a function marked with the __attribute__((destructor)) attribute will be called automatically after your main() function returns.

You can find the GCC documentation here:

constructor
destructor
The constructor attribute causes the function to be called automatically before execution enters main (). Similarly, the destructor attribute causes the function to be called automatically after main () has completed or exit () has been called. Functions with these attributes are useful for initializing data that will be used implicitly during the execution of the program.
These attributes are not currently implemented for Objective C.

Note: The GCC documentation tells that these attributes are not implemented for Objective-C. However this seems to work as expected with my tests using Clang ‘clang-703.0.31’ from Xcode 7.3.1.

 
 
Example

Here is an example of C code to demonstrate these attributes:


//
// To compile:
// clang -o constructor constructor.c
//

#include <stdio.h>
 
void constructor() __attribute__((constructor));
void destructor() __attribute__((destructor));

int main()
{
	printf ("main called\n");
	return 0;
}

void constructor()
{
	printf ("constructor called\n");
}

void destructor()
{
	printf ("destructor called\n");
}

When running this application, you will see the following output logs as you would expect:

constructor called
main called
destructor called

 
 
How does it work under the hood?

When you mark functions with these attributes, the compiler will create in your binary the sections called __mod_init_func for the constructors and __mod_term_func for the destructors. These sections contain the list of function pointers. You can use the excellent MachOView to see these sections:


ACTION

When your application is launched, dyld will call the constructors before your main() function is called. This is handled by the following dyld function:

void ImageLoaderMachO::doModInitFunctions(const LinkContext& context)

The destructors are handled by the dyld function:

void ImageLoaderMachO::doTermination(const LinkContext& context)

Since dyld is open source you can look at the implementation in the file ImageLoaderMachO.cpp of dyld. The source code for macOS 10.11.4 is available here.

 
 
Example of use

  • The first obvious usage is to be able to initialize some global variables with a constructor and do some cleanup with a destructor. It could be used to initialize some libraries too.
  •  

  • Another usage is code injection. In a previous post ‘Simple code injection using DYLD_INSERT_LIBRARIES’ I wrote code to replace some methods with an Objective-C +(void)load class method. Using a constructor would allow to inject code earlier in the process.
  •  

  • A constructor attribute could be used to implement a software protection. You could encrypt your executable with a custom encryption and use a constructor function to decrypt the binary just before it is loaded.

Blowfish operations with key size longer than 448 bits in macOS 10.11.5 / iOS 9.3.2

Posted: July 10th, 2016 | Author: | Filed under: iOS, macOS, Programming | Tags: , , , , | No Comments »

Until macOS 10.11.4 and iOS 9.3.1 CommonCrypto/corecrypto supported Blowfish operations with key sizes longer than 448 bits. Starting with macOS 10.11.5 and iOS 9.3.2 this is no longer the case: the minimum and maximum key sizes are now enforced (respectively kCCKeySizeMinBlowfish 8 bytes and kCCKeySizeMaxBlowfish 56 bytes).

This is probably the fix for CVE-2016-1802:

ACTION

If you perform a Blowfish operation with a key length longer than 448 bits, it will now fail with an error kCCParamError. Below is an example of code using a 64 bytes Blowfish key that works on macOS 10.11.4 / iOS 9.3.1 but returns an error kCCParamError on newer systems:


#import <Foundation/Foundation.h>
#import <CommonCrypto/CommonCrypto.h>

int main(int argc, const char * argv[])
{
	@autoreleasepool
	{
		uint8_t keyData[64] =
		{
			0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF,
            0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF,
			0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF,
			0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF
		};
		
		CCCryptorRef cryptorRef;
		CCCryptorStatus status = CCCryptorCreate(kCCDecrypt, kCCAlgorithmBlowfish, 0, keyData, sizeof(keyData), NULL, &cryptorRef);
		
		NSLog(@"CCCryptorCreate result: %d", status);
		if(status != kCCSuccess)
		{
			NSLog(@"*** CCCryptorCreate failed!!!");
		}
	}
	
    return 0;
}

If you have to support Blowfish with a key longer than 448 bits, you can’t use anymore CommonCrypto and should switch to a different implementation. Note that using Blowfish with a key longer than 448 bits is not recommended as it weakens the security guaranteed by the algorithm.


State Preservation and Restoration Debug Logs

Posted: July 8th, 2016 | Author: | Filed under: Debugging, iOS, macOS, Programming | Tags: , , , , , | No Comments »

The State Preservation and Restoration system is well documented here:
Preserving Your App’s Visual Appearance Across Launches.

But what is not well known is that there is a secret preference to enable debug logs. You can set the preference UIStateRestorationDebugLogging to YES in your main function before the call to UIApplicationMain:

[[NSUserDefaults standardUserDefaults] setBool:YES forKey:@"UIStateRestorationDebugLogging"];

There is also a less useful ‘Developer Mode’ secret preference which will skip the deletion of the restoration archive when the app crashes. To enable this mode set the preference UIStateRestorationDeveloperMode to YES in your main function before the call to UIApplicationMain:

[[NSUserDefaults standardUserDefaults] setBool:YES forKey:@"UIStateRestorationDeveloperMode"];