Configuring Host-Based Intrusion Detection

When to Use

Use this skill when:

• Deploying HIDS agents (Wazuh, OSSEC, AIDE) across Windows and Linux endpoints
• Configuring file integrity monitoring (FIM) for compliance (PCI DSS 11.5, NIST SI-7)
• Monitoring system configuration changes, rootkit detection, and security policy violations
• Integrating HIDS alerts with SIEM platforms for centralized monitoring

Do not use this skill for network-based IDS (Suricata, Snort) or for EDR deployment.

Prerequisites

• Wazuh server (manager) deployed and accessible from endpoints
• Administrative access to target endpoints
• Network connectivity: agents to Wazuh manager on port 1514 (TCP/UDP) and 1515 (TCP enrollment)
• Wazuh dashboard (OpenSearch Dashboards) for alert visualization
• Understanding of critical files/directories to monitor per OS

Workflow

Step 1: Install Wazuh Agent

Windows:

# Download and install Wazuh agent
Invoke-WebRequest -Uri "https://packages.wazuh.com/4.x/windows/wazuh-agent-4.9.0-1.msi" `
  -OutFile "wazuh-agent.msi"
msiexec /i wazuh-agent.msi /q WAZUH_MANAGER="wazuh-manager.corp.com" `
  WAZUH_REGISTRATION_SERVER="wazuh-manager.corp.com" WAZUH_AGENT_GROUP="windows-workstations"
net start WazuhSvc

Linux (Debian/Ubuntu):

curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor -o /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" \
  > /etc/apt/sources.list.d/wazuh.list
apt-get update && apt-get install wazuh-agent -y
sed -i 's/MANAGER_IP/wazuh-manager.corp.com/' /var/ossec/etc/ossec.conf
systemctl daemon-reload && systemctl enable --now wazuh-agent

Step 2: Configure File Integrity Monitoring (FIM)

Edit agent configuration (

/var/ossec/etc/ossec.conf

or

C:\Program Files (x86)\ossec-agent\ossec.conf

):

<syscheck>
  <!-- Scan frequency: every 12 hours -->
  <frequency>43200</frequency>
  <scan_on_start>yes</scan_on_start>
  <alert_new_files>yes</alert_new_files>

  <!-- Linux critical directories -->
  <directories check_all="yes" realtime="yes">/etc</directories>
  <directories check_all="yes" realtime="yes">/usr/bin</directories>
  <directories check_all="yes" realtime="yes">/usr/sbin</directories>
  <directories check_all="yes" realtime="yes">/bin</directories>
  <directories check_all="yes" realtime="yes">/sbin</directories>
  <directories check_all="yes">/boot</directories>

  <!-- Windows critical directories -->
  <directories check_all="yes" realtime="yes">C:\Windows\System32</directories>
  <directories check_all="yes" realtime="yes">C:\Windows\SysWOW64</directories>
  <directories check_all="yes" realtime="yes">%PROGRAMFILES%</directories>

  <!-- Windows registry monitoring -->
  <windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run</windows_registry>
  <windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce</windows_registry>
  <windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services</windows_registry>

  <!-- Ignore frequently changing files -->
  <ignore>/etc/mtab</ignore>
  <ignore>/etc/resolv.conf</ignore>
  <ignore type="sregex">.log$</ignore>
</syscheck>

Step 3: Configure Rootkit Detection

<rootcheck>
  <disabled>no</disabled>
  <frequency>43200</frequency>
  <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
  <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
  <system_audit>/var/ossec/etc/shared/system_audit_rcl.txt</system_audit>
  <check_dev>yes</check_dev>
  <check_files>yes</check_files>
  <check_if>yes</check_if>
  <check_pids>yes</check_pids>
  <check_ports>yes</check_ports>
  <check_sys>yes</check_sys>
  <check_trojans>yes</check_trojans>
  <check_unixaudit>yes</check_unixaudit>
</rootcheck>

Step 4: Configure Log Analysis Rules

<!-- Custom rules in /var/ossec/etc/rules/local_rules.xml -->
<group name="local,syscheck,">
  <!-- Alert on critical binary modifications -->
  <rule id="100001" level="12">
    <if_sid>550</if_sid>
    <match>/usr/bin/|/usr/sbin/|/bin/|/sbin/</match>
    <description>Critical system binary modified: $(file)</description>
    <group>syscheck,pci_dss_11.5,</group>
  </rule>

  <!-- Alert on new executable in temp directories -->
  <rule id="100002" level="10">
    <if_sid>554</if_sid>
    <match>/tmp/|/var/tmp/</match>
    <description>New file created in temp directory: $(file)</description>
    <group>syscheck,malware,</group>
  </rule>

  <!-- Alert on SSH configuration changes -->
  <rule id="100003" level="10">
    <if_sid>550</if_sid>
    <match>/etc/ssh/sshd_config</match>
    <description>SSH configuration modified</description>
    <group>syscheck,authentication,</group>
  </rule>
</group>

Step 5: Configure Active Response

<!-- Auto-block IP after repeated authentication failures -->
<active-response>
  <command>firewall-drop</command>
  <location>local</location>
  <rules_id>5712</rules_id>
  <timeout>600</timeout>
</active-response>

<!-- Disable account after brute force detection -->
<active-response>
  <disabled>no</disabled>
  <command>disable-account</command>
  <location>local</location>
  <rules_id>100100</rules_id>
  <timeout>3600</timeout>
</active-response>

Step 6: Integrate with SIEM

# Wazuh to Splunk via Filebeat
# Edit /etc/filebeat/filebeat.yml:
filebeat.inputs:
  - type: log
    paths:
      - /var/ossec/logs/alerts/alerts.json
    json.keys_under_root: true
output.elasticsearch:
  hosts: ["https://splunk-hec:8088"]

# Wazuh to Elastic via direct integration
# Wazuh indexer feeds directly into OpenSearch/Elasticsearch
# Dashboard: https://wazuh-dashboard:5601

Key Concepts

Term	Definition
HIDS	Host-based Intrusion Detection System; monitors individual endpoints for malicious activity
FIM	File Integrity Monitoring; detects unauthorized changes to files by comparing cryptographic hashes
Syscheck	Wazuh/OSSEC module for file integrity monitoring and registry monitoring
Rootcheck	Wazuh/OSSEC module for rootkit and malware detection
Active Response	Automated defensive action triggered by HIDS alert (IP block, account disable)
CDB List	Constant Database list used for custom lookups in Wazuh rules

Tools & Systems

• Wazuh: Open-source HIDS platform (fork of OSSEC) with manager, agent, and dashboard
• OSSEC: Original open-source HIDS (predecessor to Wazuh)
• AIDE (Advanced Intrusion Detection Environment): Standalone file integrity checker for Linux
• Tripwire: Commercial file integrity monitoring solution
• Samhain: Open-source HIDS focused on file integrity and log monitoring

Common Pitfalls

• Monitoring too many directories: FIM on entire filesystems generates excessive alerts. Focus on critical system binaries, configuration files, and web roots.
• Not excluding noisy files: Frequently changing files (logs, temp, caches) generate false positive FIM alerts. Maintain exclusion lists.
• Ignoring baseline establishment: First FIM scan creates a baseline. Changes detected before baseline stabilization are noise, not threats. Allow 48 hours for baseline.
• Active response without testing: Auto-blocking IPs or disabling accounts can cause outages. Test active response rules in a non-production environment first.
• Agent enrollment failures: Agents must successfully enroll with the manager before monitoring begins. Verify firewall rules allow port 1514 and 1515 traffic.