Malware Analysis Patterns

Common Behaviors

Persistence Mechanisms

• Registry Run keys (Windows)
• Scheduled tasks / cron jobs
• Service installation
• DLL hijacking
• Boot sector modification

Anti-Analysis Techniques

• Debugger detection (ptrace, IsDebuggerPresent)
• VM detection (CPUID, registry, timing)
• Sandbox detection (user activity, artifacts)
• Code obfuscation
• Packed/encrypted payloads

C2 Communication

• HTTP/HTTPS beaconing
• DNS tunneling
• Custom protocols
• Domain generation algorithms (DGA)
• Dead drops (pastebin, social media)

Analysis Techniques

String Decryption

XOR decryption:

def xor_decrypt(data, key):
    return bytes([b ^ key[i % len(key)] for i, b in enumerate(data)])

# Single byte key
decrypted = xor_decrypt(encrypted, b'\x42')

# Multi-byte key
decrypted = xor_decrypt(encrypted, b'secret')

RC4 decryption:

def rc4_init(key):
    S = list(range(256))
    j = 0
    for i in range(256):
        j = (j + S[i] + key[i % len(key)]) % 256
        S[i], S[j] = S[j], S[i]
    return S

def rc4_crypt(S, data):
    S = S.copy()
    i = j = 0
    result = []
    for byte in data:
        i = (i + 1) % 256
        j = (j + S[i]) % 256
        S[i], S[j] = S[j], S[i]
        result.append(byte ^ S[(S[i] + S[j]) % 256])
    return bytes(result)

Payload Extraction

LD_PRELOAD hooking:

// hook_memfd.c
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

int memfd_create(const char *name, unsigned int flags) {
    int (*real_memfd)(const char*, unsigned int) = dlsym(RTLD_NEXT, "memfd_create");
    int fd = real_memfd(name, flags);
    fprintf(stderr, "[HOOK] memfd_create('%s') = %d\n", name, fd);
    return fd;
}

ssize_t write(int fd, const void *buf, size_t count) {
    ssize_t (*real_write)(int, const void*, size_t) = dlsym(RTLD_NEXT, "write");
    // Dump to file for analysis
    FILE *f = fopen("/tmp/captured_payload", "ab");
    fwrite(buf, 1, count, f);
    fclose(f);
    return real_write(fd, buf, count);
}

Compile and use:

gcc -shared -fPIC -o hook.so hook.c -ldl
LD_PRELOAD=./hook.so ./malware

Memory Forensics

Volatility 3 basics:

# List processes
vol -f memory.dmp windows.pslist

# Dump process memory
vol -f memory.dmp windows.memmap --pid 1234 --dump

# Find injected code
vol -f memory.dmp windows.malfind

Indicators of Compromise (IOCs)

Network IOCs

• IP addresses
• Domain names
• URLs
• User-Agent strings
• JA3/JA3S fingerprints

Host IOCs

• File hashes (MD5, SHA256)
• File paths
• Registry keys
• Mutex names
• Service names

Documentation Template

## IOCs

### Network
| Type | Value | Context |
|------|-------|---------|
| IP | 192.168.1.100 | C2 server |
| Domain | evil.com | DGA generated |

### Host
| Type | Value | Context |
|------|-------|---------|
| Hash | abc123... | Main payload |
| Path | /tmp/.hidden | Dropped file |
| Mutex | Global\XYZ | Infection marker |

Anti-Debug Bypass

ptrace Detection

// Malware checks:
if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) {
    exit(1);  // Being debugged
}

// GDB bypass:
(gdb) catch syscall ptrace
(gdb) commands
> set $rax = 0
> continue
> end

Timing Checks

// Malware checks:
t1 = rdtsc();
// ... code ...
t2 = rdtsc();
if (t2 - t1 > threshold) exit(1);  // Debugger detected

// Bypass: Patch comparison or NOP the check