Crypto in Malware Analysis

This skill helps you identify cryptographic and compression algorithms in malware binaries and recognize packing techniques during reverse engineering.

When to Use This Skill

Use this skill when:

• You're analyzing a binary and suspect encryption or compression
• You see unusual patterns in assembly (lots of XORs, shifts, rotates)
• Static analysis seems misleading or incomplete
• You need to identify packing algorithms
• You're investigating obfuscated malware
• You encounter Windows crypto API calls

Identifying Cryptographic/Compression Algorithms

Technique-First Heuristics

Look for these patterns in assembly code:

1. Shift/Rotate Heavy Code: Lots of shifts, rotates, XORs, and 32-bit arithmetic in tight loops often indicate cryptographic operations

2. Lookup Tables: Check

   .data

   section or runtime-generated tables for S-boxes or other crypto tables

3. RC4 Indicators: Repeating loops of 256 iterations (0x100) strongly suggest RC4

Windows Crypto/Compression APIs

CryptDeriveKey / CryptCreateHash

When you see these API calls, examine the second parameter - it's an

ALG_ID

that identifies the algorithm:

• Check the ALG_ID value against Microsoft's documentation
• Common values indicate specific algorithms (DES, RC4, SHA, MD5, etc.)
• Reference: https://learn.microsoft.com/en-us/windows/win32/seccrypto/alg-id

RtlCompressBuffer / RtlDecompressBuffer

These calls indicate built-in Windows compression:

• LZNT1
• XPRESS
• Other Windows compression formats

Constants and Tables Fingerprinting

You can often identify algorithms by searching for constants or table values:

1. Extract the first dword of suspicious tables
2. Search online for known crypto constants
3. Compare against known algorithms like AES tables

Example: AES has well-known S-box and inverse S-box tables that can be fingerprinted.

RC4 Recognition Checklist

RC4 is commonly used in malware and has distinctive patterns:

1. Two loops of 256 iterations (initialization + Key Scheduling Algorithm)
2. PRGA loop that uses modulo 256 operations
3. XOR keystream with data in the final loop

Look for this pattern:

Loop 1: 256 iterations (init)
Loop 2: 256 iterations (KSA)
Loop 3: PRGA with % 256 and XOR

Unpacking Binaries

Understanding Packers

Packers transform binaries to mislead static analysis by:

• Adding junk code
• Encrypting sections
• Performing runtime unpacking

Your goal: Catch the moment the binary:

1. Allocates/decrypts real code in memory
2. Marks it executable
3. Jumps into the unpacked code

Identifying Packed Binaries

Look for these indicators:

1. Lack of strings or only packer-specific strings
2. Many strings without cross-references (common in commercial packers)
3. Use packer-ID tools:
   • PEiD
   • Exeinfo PE

Unpacking Strategy

Follow this approach:

1. Start from the bottom and work upward - unpackers often jump late in execution

2. Look for jump patterns:

   • JMP/CALL reg

     patterns
   • Stack tricks like

     push addr; retn

3. Set breakpoints on memory operations:

   • VirtualAlloc

     - watch for code allocation

   • VirtualProtect

     - watch for permission changes to RWX
   • Track RWX regions that appear during execution

4. Watch for string explosion - a sudden appearance of many strings after a jump often indicates you've reached unpacked code

5. Dump and fix:

   • Dump the memory at the right moment
   • Fix headers with tools like PE-bear
   • Verify the dumped binary runs correctly

Analysis Workflow

When approaching a potentially packed/encrypted binary:

1. Initial reconnaissance

   • Run PEiD/Exeinfo PE to check for known packers
   • Look at section characteristics (entropy, permissions)
   • Check for unusual import tables

2. Static analysis

   • Look for crypto API calls
   • Search for known constants/tables
   • Identify suspicious loop patterns

3. Dynamic analysis

   • Set breakpoints on VirtualAlloc/VirtualProtect
   • Watch for RWX memory regions
   • Monitor for string appearance

4. Unpacking

   • Dump memory at the right moment
   • Fix headers
   • Verify functionality

Common Pitfalls

• Don't trust static analysis alone on packed binaries
• Don't assume all encryption is malicious (legitimate software uses it too)
• Don't forget to check for anti-debugging techniques
• Don't skip the header fixing step after dumping

Tools to Have Ready

• Packer identification: PEiD, Exeinfo PE
• Memory dumping: Scylla, x64dbg with plugins
• Header fixing: PE-bear, CFF Explorer
• Disassembly: IDA Pro, Ghidra, x64dbg
• Dynamic analysis: x64dbg, WinDbg