Dynamic Malware Analysis: Behavioral Analysis in a Controlled Environment

Dynamic malware analysis executes a suspicious sample in a controlled environment and observes its runtime behavior. Where static analysis tells you what the code contains, dynamic analysis tells you what it actually does. Dynamic analysis is indispensable when dealing with heavily obfuscated, packed, or polymorphic malware that static techniques cannot fully characterize. The goal is to capture every artifact — processes, files, registry changes, network connections — produced during execution while maintaining isolation from production systems.

Building an Isolated Analysis Environment

Hypervisor and Network Configuration

The analysis environment must be completely isolated from production networks but may need limited internet connectivity to trigger C2 behavior.

# Recommended architecture:
# Host: Bare-metal or ESXi/vSphere/KVM
# Analysis VM: FlareVM (Windows) or REMnux (Linux)
# Monitoring VM: REMnux (network monitoring)
# Network: Host-only or isolated virtual switch

# VMware network configurations:
# Host-Only: VM can only talk to host — good for total isolation
# Custom VMnet: Create isolated segment between analysis VMs
# NAT with INetSim: VM gets simulated internet responses

# VirtualBox: File > Host Network Manager > Create
# Assign VMs to the isolated host-only network

FlareVM Setup (Windows Analysis)

# FlareVM is a Windows-based malware analysis distribution
# Start with a clean Windows 10/11 VM (licensed or evaluation)
# Download FlareVM from: https://github.com/mandiant/flare-vm

# In PowerShell (as Administrator):
(New-Object net.webclient).DownloadFile('https://raw.githubusercontent.com/mandiant/flare-vm/main/install.ps1','install.ps1')
Unblock-File .\install.ps1
Set-ExecutionPolicy Unrestricted -Force
.\install.ps1

# Included tools (100+):
# - x64dbg, OllyDbg, WinDbg
# - IDA Free, Ghidra, Binary Ninja
# - Process Monitor, Process Hacker
# - Wireshark, Fiddler
# - PE-Bear, CFF Explorer
# - Malcat, FLOSS, YARA
# - Python, Perl, Java, Ruby

# Take a clean snapshot BEFORE any malware execution

REMnux Setup (Linux/Network Monitoring)

# REMnux is a Linux distribution for malware analysis
# Download from: https://remnux.org

# As network monitor:
# Install on a separate VM with two network interfaces
# Interface 1: Management (to host)
# Interface 2: Connected to analysis network

# Configure INetSim (internet simulator) on REMnux
# Edit /etc/inetsim/inetsim.conf:
service_bind_address  192.168.100.1  # REMnux analysis IP

# Services simulated by INetSim:
# HTTP/HTTPS, FTP, DNS, SMTP, POP3, IRC, and more
# Malware gets fake responses — allows full behavioral capture

# Start INetSim
sudo inetsim

# Configure analysis VM to use REMnux as DNS and default gateway
# DNS: 192.168.100.1 (REMnux IP)
# Gateway: 192.168.100.1

Pre-Execution Baseline Collection

Before running the sample, capture the current system state. Comparison against this baseline reveals all changes made by the malware.

# Windows baseline commands (run before execution)
# Process list
tasklist /v > baseline_processes.txt

# Network connections
netstat -ano > baseline_netstat.txt

# Registry snapshot (key areas)
reg export HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run baseline_run.reg
reg export HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run baseline_run_user.reg
reg export HKLM\SYSTEM\CurrentControlSet\Services baseline_services.reg

# Scheduled tasks
schtasks /query /fo CSV /v > baseline_tasks.csv

# Startup entries
wmic startup list full > baseline_startup.txt

# Installed services
sc query state= all > baseline_services.txt

# File system snapshot (key directories)
dir /s /a C:\Windows\System32 > baseline_system32.txt
dir /s /a C:\Users\%USERNAME%\AppData\Roaming > baseline_appdata.txt

Process Monitor (ProcMon)

Sysinternals Process Monitor captures every file system, registry, network, and process event in real-time. It is the most information-dense Windows monitoring tool available.

ProcMon Configuration for Malware Analysis

# Start Process Monitor (ProcMon.exe) as Administrator
# Before execution:
# Filter > Add Filter: Process Name > contains > malware.exe
# (Or monitor all and filter afterward)

# Capture all 5 event types:
# - Registry (HKLM, HKCU access)
# - File System (creates, writes, deletes)
# - Network (connections)
# - Process/Thread (creation, injection)
# - Profiling events

# After execution, apply filters:
# Filter: Process Name > contains > malware.exe
# Filter: Operation > contains > WriteFile
# Filter: Operation > contains > RegSetValue
# Filter: Operation > contains > Process Create

Key ProcMon Indicators

# Save ProcMon log
# File > Save > All events (PML format)
# File > Save > As CSV for external analysis

# Useful ProcMon filters:
# Path contains: AppData\Roaming  -- common dropper location
# Path contains: Startup          -- autostart persistence
# Path contains: Temp             -- temporary malware files
# Result: SUCCESS                 -- only successful operations
# Category: Write                 -- all write operations
# Operation: Process Create       -- child process spawning

Process Hacker / Process Explorer

# Process Hacker (open source, more features than Task Manager)
# Download: https://processhacker.sourceforge.io

# Key views during malware execution:
# - Process tree: See parent-child relationships
# - Memory: View injected memory regions
#   - Look for executable regions with no file backing
#   - Right-click > Memory > Filter: Protection = Execute
# - Handles: Open files, registry keys, mutexes, pipes
# - DLLs: Loaded modules (compare against baseline)
# - Network: Active connections per process
# - Services: Check for new services

# Identify process hollowing:
# - Legitimate process (svchost.exe) with unusual memory regions
# - Memory section has Execute+Write (WX) — never normal for legitimate code
# - Image base doesn't match expected PE image base

# In Process Hacker: Right-click process > Properties > Memory
# Look for memory regions with protection: RWX or RX with no mapped file

Wireshark Network Monitoring During Execution

# Start Wireshark capture on analysis VM interface before execution
# Capture filter (to limit noise):
not arp and not mdns and not llmnr

# After execution, apply display filters:

# All outbound connections (non-local)
ip.dst != 192.168.100.0/24 and not ip.dst == 255.255.255.255

# DNS queries made by malware
dns

# HTTP/HTTPS
http or tls

# C2 protocol analysis — look for:
# Periodic beaconing (uniform time intervals)
# Unusual ports for HTTP (8080, 8443, 4444)
# Domain generation algorithm (DGA) patterns: random-looking domains
# Long DNS query names (DNS tunneling)
# ICMP with data payload (ICMP tunneling)

# Capture C2 certificate information
tls.handshake.type == 11    # Certificate message
# Extract certificate fields:
tls.handshake.certificate

API Monitor — System Call Interception

API Monitor hooks into the Windows API and logs every function call with arguments and return values in real-time.

# API Monitor (http://www.rohitab.com/apimonitor)
# Configure monitoring before execution:
# 1. File > Monitor New Process > Browse to malware.exe
# 2. Select API categories to monitor:
#    - Windows > Security > Cryptography
#    - Windows > System Services > Process and Thread
#    - Windows > Networking > Windows Sockets 2
#    - Windows > File System > File Management

# Key APIs to watch:
# Process injection indicators:
VirtualAllocEx    -- Allocate memory in remote process
WriteProcessMemory -- Write payload to remote process
CreateRemoteThread -- Execute payload in remote process
NtCreateThreadEx   -- Modern alternative to CreateRemoteThread

# Persistence:
RegCreateKeyEx     -- Create registry key
RegSetValueEx      -- Set registry value
CreateService      -- Install new service

# Network:
WSAConnect / connect  -- TCP connections (check destination)
HttpSendRequest       -- HTTP C2 communication
InternetReadFile      -- Download payloads

# Crypto (decryption of payloads):
CryptDecrypt / CryptImportKey  -- Decrypting embedded data
RtlDecompressBuffer            -- Decompressing data

# Capture API call log:
# File > Save Monitored Processes

Registry Analysis

# Compare registry before/after using RegShot (free tool)
# RegShot: 1st shot before, 2nd shot after, compare

# Manual comparison (PowerShell):
# Before:
Get-ChildItem -Path HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Run | Export-Csv before.csv
# After:
Get-ChildItem -Path HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Run | Export-Csv after.csv
# Compare:
Compare-Object (Import-Csv before.csv) (Import-Csv after.csv)

# Key persistence locations:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce
HKLM\SYSTEM\CurrentControlSet\Services
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon (Userinit)
HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options (debugger hijack)
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\BootExecute

Detecting Sandbox Evasion Techniques

Sophisticated malware detects when it's running in a sandbox and either terminates, alters behavior, or sleeps indefinitely. Understanding these techniques allows analysts to defeat them.

Time-Based Evasion

# Malware sleeps longer than sandbox analysis window (usually 3-10 minutes)
# Techniques:
Sleep(600000)    -- 10 minutes sleep
GetTickCount()   -- Check if system has been running long enough

# Bypass: Patch Sleep() calls or accelerate VM clock
# In x64dbg: Set breakpoint on Sleep, when hit, set argument to 0 or skip call
# Or: Use Process Hacker to accelerate process time

# Speedrun approach: Patch kernel32.Sleep and NTDLL.ZwDelayExecution to return immediately
# In x64dbg:
# Breakpoint on Sleep
# When triggered: Right-click ESP → Modify register → Set arg to 1

VM/Sandbox Detection

# Common VM artifacts malware checks:
# Registry keys:
HKLM\SOFTWARE\VMware, Inc.\VMware Tools   -- VMware
HKLM\SOFTWARE\Oracle\VirtualBox Guest Additions  -- VirtualBox
HKLM\HARDWARE\ACPI\DSDT\VBOX__            -- VirtualBox ACPI

# Process names:
vmtoolsd.exe, vmwaretray.exe   -- VMware
vboxservice.exe, vboxtray.exe  -- VirtualBox
sandboxie.exe, cuckoo.exe      -- Sandbox tools
procmon.exe, wireshark.exe     -- Analysis tools

# Hardware checks:
MAC address: 00:0C:29:xx → VMware, 08:00:27:xx → VirtualBox
CPUID instruction reveals hypervisor
Low disk space (< 100GB) suggests sandbox
Low RAM (< 4GB) suggests sandbox
CPU core count = 1 suggests sandbox

# Bypass techniques:
# Change MAC address to a real manufacturer
# Modify registry to remove VM-specific keys
# Use VMware with tools removed
# Increase RAM and vCPU count
# Pafish checker: shows what VM artifacts are detectable
# https://github.com/a0rtega/pafish

User Interaction Checks

# Malware checks for human presence:
GetCursorPos()          -- mouse must have moved
GetForegroundWindow()   -- active window must exist
GetLastInputInfo()      -- time since last user input
GetKeyboardState()      -- keys must have been pressed

# Bypass: Use a script to simulate user activity during analysis
# AutoIt or PowerShell to move mouse and type keys

# Screen resolution check
GetSystemMetrics(SM_CXSCREEN) -- common sandbox: 800x600 or 1024x768
# Set sandbox resolution to 1920x1080

# Running process count check (sandboxes have few processes)
CreateToolhelp32Snapshot + Process32First/Next
# Bypass: Run many benign processes before malware execution

Automated Sandbox Analysis

ANY.RUN (Interactive Sandbox)

# ANY.RUN at https://app.any.run (free tier available)
# Features:
# - Interactive: You can click/type inside the VM
# - Real-time network capture
# - Process tree visualization
# - Automatic IOC extraction
# - Suricata IDS signatures applied
# - Community sharing of analysis reports

# Useful for: Malware that requires user interaction (clicking OK dialogs)
# Limitation: Malware may detect any.run environment

Cuckoo Sandbox (Self-Hosted)

# Cuckoo is the open-source self-hosted sandbox
# Modern fork: https://github.com/cuckoosandbox/cuckoo

# Installation (Ubuntu 20.04)
pip install cuckoo
cuckoo init
cuckoo web --host 0.0.0.0 --port 8080

# Submit sample via CLI
cuckoo submit /path/to/malware.exe
cuckoo submit --url http://malicious.example.com

# Submit with options
cuckoo submit --timeout 120 --package exe --options procmemdump=yes malware.exe

# Cuckoo generates:
# - Full behavioral report (JSON)
# - Network PCAP
# - Memory dumps
# - Screenshots
# - Suricata alerts
# - Extracted dropped files
# - Malware signatures matched

Hybrid Analysis and MalwareBazaar

# Hybrid Analysis (https://www.hybrid-analysis.com)
# Free sandbox with detailed reports
# API access available:
curl -X POST "https://www.hybrid-analysis.com/api/v2/submit/file" \
  -H "api-key: YOUR_API_KEY" \
  -F "[email protected]" \
  -F "environment_id=120"   # 120=Windows 10 64-bit

# Check report
curl "https://www.hybrid-analysis.com/api/v2/report/JOB_ID/summary" \
  -H "api-key: YOUR_API_KEY"

Post-Execution Analysis Workflow

# 1. Stop all monitoring tools
# 2. Save all logs before any cleanup
# 3. Document timeline:
#    - T+0: Malware execution
#    - T+5s: DNS query to malicious.domain
#    - T+10s: HTTP GET /stage2.bin
#    - T+15s: Process hollowing of svchost.exe
#    - T+20s: Registry run key added

# 4. Extract IOCs from all sources:
# Network: Wireshark PCAP
# Process: ProcMon log, Process Hacker
# Registry: RegShot diff
# Files: ProcMon WriteFile events

# 5. Correlate with static analysis findings
# 6. Write comprehensive report including:
#    - Executive summary
#    - Technical TTPs (MITRE ATT&CK mapping)
#    - Complete IOC list (hashes, IPs, domains, registry, files)
#    - YARA rules
#    - Sigma rules for SIEM
#    - Remediation recommendations