Linux Forensics Investigation

A systematic approach to investigating Linux systems for security incidents, malware, and unauthorized access.

Investigation Workflow

Follow this sequence to ensure comprehensive coverage:

1. Preserve evidence - Set up clean environment, avoid modifying the target
2. Gather initial information - System state, processes, network connections
3. Capture memory (if live) - Use LiME before shutdown
4. Create disk image - Forensic copy with hash verification
5. Analyze artifacts - Logs, autostart locations, user accounts, file system
6. Document findings - Timeline, suspicious indicators, evidence chain

---

1. Initial Information Gathering

Set Up Clean Environment

Before touching the target system, prepare a trusted environment:

# Mount USB with known-good binaries
export PATH=/mnt/usb/bin:/mnt/usb/sbin
export LD_LIBRARY_PATH=/mnt/usb/lib:/mnt/usb/lib64

Collect System State

Run these commands to capture the current system state:

# Time and OS information
date
uname -a

# Network state
ifconfig -a || ip a
netstat -anp
netstat -rn; route

# Running processes
ps -ef
lsof -V

# System resources
df; mount
free

# User activity
w
last -Faiwx

# Kernel modules
lsmod

# User accounts
cat /etc/passwd
cat /etc/shadow

# Recently modified files
find / -type f -mtime -1 -print 2>/dev/null

Red Flags to Watch For

• Root processes with high PIDs - Normal root processes have low PIDs
• Users without shells in /etc/passwd - Check for password hashes in /etc/shadow
• Unexpected network connections - Especially outbound to unknown IPs
• Modified system binaries - Compare against known-good versions

---

2. Memory Acquisition (Live Systems)

Using LiME

LiME (Linux Memory Extractor) captures volatile memory:

# Install (if identical kernel available)
apt-get install lime-forensics-dkms

# Or compile from source with matching kernel headers
make -C /lib/modules/$(uname -r)/build M=$PWD
sudo insmod lime.ko "path=/path/to/mem_dump.bin format=lime"

Important: Do not install anything on the victim machine if possible. Use a USB with pre-compiled LiME matching the kernel version.

LiME Formats

• raw - Concatenated segments
• padded - Raw with zero-padding
• lime - Recommended, includes metadata

Network Transfer

# Send dump over network instead of local storage
sudo insmod lime.ko "path=tcp:4444 format=lime"

---

3. Disk Imaging

Shutdown Strategy

• Normal shutdown - Allows filesystem sync but malware may destroy evidence
• Pull the plug - Risk of data loss but prevents malware cleanup

If malware suspected: Run

sync

then power off immediately.

Create Forensic Image

# Basic raw copy
dd if=/dev/sdX of=/path/to/image.img bs=512

# With hash verification (recommended)
dcfldd if=/dev/sdX of=/path/to/image.img bs=512 hash=sha256 hashwindow=1M hashlog=/path/to/hashes.log

Always mount images as read-only to prevent modification.

---

4. Disk Image Analysis

Identify Image Type

# Check file type
file disk.img

# Check image format
img_stat -t evidence.img
img_stat -i list  # List supported formats

Filesystem Information

# Get filesystem metadata
fsstat -i raw -f ext4 disk.img

# List directory contents
fls -i raw -f ext4 disk.img
fls -i raw -f ext4 disk.img <inode_number>

# Extract file content
icat -i raw -f ext4 disk.img <inode_number>

---

5. Malware Detection

System File Integrity

# RedHat-based systems
rpm -Va

# Debian-based systems
dpkg --verify
apt-get install debsums
debsums | grep -v "OK$"

Search Installed Programs

# Debian package database
cat /var/lib/dpkg/status | grep -E "Package:|Status:"
cat /var/log/dpkg.log | grep installed

# RedHat RPM database
rpm -qa --root=/mntpath/var/lib/rpm

# Find executables not in package databases
find /sbin/ -exec dpkg -S {} \; | grep "no path found"
find /sbin/ -exec rpm -qf {} \; | grep "is not"

# Find all executable files
find / -type f -executable 2>/dev/null | grep <pattern>

Check Common Installation Directories

ls /usr/local /opt /usr/sbin /usr/bin /bin /sbin

---

6. Recover Deleted Binaries

If a process was executed from a deleted file:

# Navigate to process directory
cd /proc/<PID>/

# Get memory address from maps
head -1 maps

# Extract from memory
dd if=mem bs=1 skip=<address> count=<size> of=/path/to/recovered

---

7. Autostart and Persistence Locations

Scheduled Tasks

# Cron and anacron
cat /var/spool/cron/crontabs/*
cat /var/spool/cron/atjobs
cat /var/spool/anacron
cat /etc/cron*
cat /etc/at*
cat /etc/anacrontab
cat /etc/incron.d/*
cat /var/spool/incron/*

# Check 0anacron stubs for modifications
for d in /etc/cron.*; do [ -f "$d/0anacron" ] && stat -c '%n %y %s' "$d/0anacron"; done

# Search for suspicious commands in cron
find /etc/cron.* -type f -exec grep -l -E 'curl|wget|/bin/sh|python|bash -c' {} \;

SSH and Account Backdoors

# Check for root login enabled
grep -E '^\s*PermitRootLogin' /etc/ssh/sshd_config

# System accounts with interactive shells
awk -F: '($7 ~ /bin\/(sh|bash|zsh)/ && $1 ~ /^(games|lp|sync|shutdown|halt|mail|operator)$/) {print}' /etc/passwd

# Check authorized keys
cat ~/.ssh/authorized_keys
cat ~/.ssh/known_hosts

Cloud C2 Indicators

# Cloudflare tunnel processes
ps aux | grep -E '[c]loudflared|trycloudflare'
systemctl list-units | grep -i cloudflared

Service Autostart Locations

Check these directories for malicious services:

• /etc/inittab

• /etc/rc.d/

  and

  /etc/rc.boot/

• /etc/init.d/

• /etc/inetd.conf

  and

  /etc/xinetd/

• /etc/systemd/system/

• /etc/systemd/system/multi-user.target.wants/

• /usr/local/etc/rc.d/

• ~/.config/autostart/

• /lib/systemd/system/

Kernel Modules

# Loaded modules
lsmod

# Module directories
ls /lib/modules/$(uname -r)
cat /etc/modprobe.d/*
cat /etc/modprobe*
cat /etc/modprobe.conf

User Login Scripts

# System-wide
ls /etc/profile.d/
cat /etc/profile
cat /etc/bash.bashrc
cat /etc/rc.local

# User-specific
cat ~/.bashrc
cat ~/.bash_profile
cat ~/.profile
ls ~/.config/autostart/

---

8. Log Analysis

System Logs

# Main system logs (Debian)
cat /var/log/syslog
cat /var/log/auth.log

# Main system logs (RedHat)
cat /var/log/messages
cat /var/log/secure

# Authentication events
grep -iE "session opened for|accepted password|new session|not in sudoers" /var/log/auth.log

# Boot, kernel, and service logs
cat /var/log/boot.log
cat /var/log/kern.log
cat /var/log/dmesg
cat /var/log/daemon.log
cat /var/log/cron

# Failed logins
cat /var/log/faillog
cat /var/log/btmp

# Web and database logs
cat /var/log/httpd/*
cat /var/log/mysqld.log
cat /var/log/xferlog

User History Files

# Shell history
cat ~/.bash_history
cat ~/.zsh_history
ls ~/.zsh_sessions/

# Application history
cat ~/.python_history
cat ~/.mysql_history
cat ~/.viminfo
cat ~/.lesshst
cat ~/.ftp_history
cat ~/.sftp_history

# Browser history
ls ~/.mozilla/firefox/
ls ~/.config/google-chrome/

# Git history
cat ~/.gitconfig
ls .git/logs/

USB Device History

# Install usbrip
pip3 install usbrip
usbrip ids download

# Get USB history
usbrip events history

# Search by vendor/product ID
usbrip events history --pid 0002 --vid 0e0f
usbrip ids search --pid 0002 --vid 0e0f

Privilege Escalation Indicators

# Sudo configuration
cat /etc/sudoers
ls /etc/sudoers.d/

# Group memberships
cat /etc/groups
cat /etc/passwd

# Look for accounts without passwords or weak hashes
cat /etc/shadow | grep -E '^([^:]+):$|^[^:]+:!!|^([^:]+):!'

---

9. File System Analysis

Timeline and Anti-Forensics Detection

# Find setuid root files
find / -user root -perm -04000 -print 2>/dev/null

# Recent files in system directories
ls -laR --sort=time /bin
ls -laR --sort=time /sbin

# Sort by inode (timestamps can be faked, inodes cannot)
ls -lai /bin | sort -n

# Find hidden files with unusual names
find / -name ".. *" -o -name "..^G" 2>/dev/null

# Check /dev for unusual files
ls -la /dev/

# Deleted files still open
lsof +L1
lsof | grep '(deleted)'

# Inode pressure check
df -i

Inode Analysis

# Find all names pointing to one inode
find / -xdev -inum <inode_number> 2>/dev/null

# Inspect inode metadata (EXT filesystem)
sudo debugfs -R "stat <inode_number>" /dev/sdX

Key inode fields:

• Links: 0 means no directory entry references this inode
• dtime: Deletion timestamp
• ctime/mtime: Metadata and content modification times

Filesystem Version Comparison

# Find new files
git diff --no-index --diff-filter=A path/to/old/ path/to/new/

# Find modified files
git diff --no-index --diff-filter=M path/to/old/ path/to/new/ | grep -E '^\+' | grep -v "Installed-Time"

# Find deleted files
git diff --no-index --diff-filter=D path/to/old/ path/to/new/

---

10. Documentation and Reporting

Evidence Chain

Document:

1. Case information - Date, time, investigator, case number
2. System information - Hostname, IP, OS version, kernel
3. Acquisition details - Tools used, hash values, timestamps
4. Findings - Suspicious files, processes, network connections
5. Timeline - Chronological sequence of events
6. Recommendations - Remediation steps

Hash Verification

Always verify evidence integrity:

# Calculate hashes
sha256sum /path/to/evidence
md5sum /path/to/evidence

# Verify against known values
sha256sum -c hashes.txt

---

Quick Reference Scripts

Use the bundled scripts for common tasks:

• scripts/gather_system_info.sh

  - Collect initial system state

• scripts/check_autostart.sh

  - Scan all persistence locations

• scripts/analyze_logs.sh

  - Extract key log entries

• scripts/find_suspicious.sh

  - Identify potential malware indicators

---

Safety Reminders

• Never modify the target system - Work from forensic images
• Verify tool integrity - Use known-good binaries from USB
• Document everything - Maintain chain of custody
• Preserve timestamps - Note when each action was performed
• Verify hashes - Ensure evidence integrity throughout investigation