I recently stumbled upon an interesting behaviour of the PE loader. Before releasing a new Hyperion version, I usually test it with several executables on different Windows platforms. In my old XP VM, everything went fine. Therefore, I started a Windows 7 instance and encrypted calc.exe with the following command:
hyperion -k 2 -s 2 -l c:\windows\system32\calc.exe test.exe
The options k and s reduce our key space to four and speed up the brute forcing process. Due to l, a log file is generated upon execution of test.exe. Unfortunatly, the calculator doesn't show up when we double click on the encrypted file. Log.txt shows the following results:
Processing Import Directory: COMCTL32.dll Name: ImageList_Destroy Name: ImageList_Add Name: ImageList_Create Ordinal: 0000019D Name: CreatePropertySheetPageW Name: PropertySheetW Ordinal: 0000017C Error
According to the log file, an API couldn't been loaded. It is located in the comctl32.dll and has the ordinal 0x17c. This is kinda strange. Therefore, I started LordPE and took a look into the export table of comctl32.dll. Indeed, there is no API which uses the ordinal 0x17c. I also analyzed the import table of calc.exe and found the entry 0x17c. So there must be a difference between Hyperion (which uses LoadLibrary()) and the Windows PE-Loader. As a next step, I started OllyDbg. My goal was to compare the memory maps of calc.exe and test.exe:
According to both memory maps, two different versions of comctl32.dll have been loaded. The packed executable gets 5.82 from a LoadLibrary() call. The unmodified calc.exe uses the PE loader and receives comctl32.dll version 6.10.
There seems the be a difference between LoadLibrary() and the Windows PE-loader. LoadLibrary() maps older versions of some system libraries into memory which can cause trouble because of missing or wrong API ordinals. I haven't yet found out the reason for this behaviour. The import table of PE files does not provide version informations at all. If anybody knows, please let me know.