Malware Analysis — Part 2: Setting Up the Windows Analysis VM with FLARE-VM

Previous: Part 1 — Building the Lab

What Is FLARE-VM?

FLARE-VM (FireEye Labs Advanced Reverse Engineering Virtual Machine) is an open source Windows-based security distribution maintained by Mandiant (now part of Google). It is, in practical terms, a PowerShell script that transforms a clean Windows installation into a fully equipped malware analysis and reverse engineering workstation — automatically installing and configuring dozens of the tools that professional analysts use every day.

Before FLARE-VM existed, setting up a Windows analysis environment meant manually downloading, installing, and configuring 30+ individual tools, each with its own configuration quirks. FLARE-VM automates all of this and ensures a consistent, reproducible environment.

Why not just use REMnux for everything?

REMnux is a Linux-based distribution. Many malware samples are designed to run specifically on Windows — they use Windows APIs, write to Windows registry keys, spawn Windows processes. To properly analyze Windows malware dynamically (meaning: running it and watching what it does), you need a Windows environment. FLARE-VM gives you that Windows environment, pre-configured for analysis.

Free or paid? FLARE-VM is completely free and open source: github.com/mandiant/flare-vm

The professional equivalent in large organizations is often a proprietary custom image built by an internal security team, with licensed tools like IDA Pro pre-installed. FLARE-VM covers everything you need to get started.

Installing a Clean Windows VM

FLARE-VM requires a clean Windows 10 or Windows 11 installation. Microsoft provides free evaluation ISOs that work perfectly for this.

Download a Windows evaluation ISO:

• Windows 10 Enterprise 90-day evaluation: microsoft.com/evalcenter
• Windows 11 Enterprise 90-day evaluation: same site, search Windows 11

After 90 days the VM will display watermarks and eventually shut down every hour — at that point, simply restore your baseline snapshot (which you'll create before the 90 days) or reinstall from the same free ISO.

Create the VM in Hyper-V

In Hyper-V Manager: New → Virtual Machine

After creation, before first boot:

• VM Settings → Network Adapter → Internal-Lab (connect to the isolated lab switch, not the internet)
• VM Settings → Security → Disable Secure Boot (required for FLARE-VM)
• VM Settings → Checkpoints → Disable automatic checkpoints (you'll manage snapshots manually)

Install Windows normally. During setup:

• Choose Windows 10/11 Enterprise (not Home)
• When asked for a product key, click "I don't have a product key"
• Create a local account (do not sign in with a Microsoft account — this is an isolated analysis machine, not a personal computer)
• Username suggestion: analyst or malware

Pre-FLARE-VM Configuration

Before running FLARE-VM, configure Windows to make the environment safe and effective for malware analysis.

Disable Windows Defender

Windows Defender will interfere with malware analysis — it will quarantine samples before you can analyze them, delete tools that security software flags, and generally prevent you from doing your job. Disable it.

Open PowerShell as Administrator:

# Disable real-time monitoring
Set-MpPreference -DisableRealtimeMonitoring $true

# Disable cloud-delivered protection
Set-MpPreference -MAPSReporting Disabled

# Disable automatic sample submission (CRITICAL)
# This would upload your malware samples to Microsoft
Set-MpPreference -SubmitSamplesConsent NeverSend

# Disable Windows Defender completely
Set-MpPreference -DisableIOAVProtection $true

Alternatively, in Windows Security → Virus & threat protection → Manage settings, toggle off all protections. You may need to also disable Tamper Protection before the toggles become editable.

Disable Windows Update

Automatic updates can trigger during an analysis session, consuming resources and potentially interfering with your observations. Disable them:

# Stop and disable the Windows Update service
Stop-Service -Name wuauserv
Set-Service -Name wuauserv -StartupType Disabled

Configure Time Zone

Set the VM time zone to a neutral time zone for analysis. This prevents local time from affecting time-based malware triggers:

Set-TimeZone -Id "UTC"

Take a Pre-Tool Snapshot

Before installing FLARE-VM, take a snapshot of this minimal clean Windows state. This gives you a fast restore point if something goes wrong during installation:

In Hyper-V Manager → right-click the VM → Checkpoint Name: Windows-Clean-PreFLARE

Installing FLARE-VM

Prerequisites

FLARE-VM requires:

• PowerShell 5+ (included in Windows 10/11)
• .NET Framework 4.0+
• The VM must have internet access during installation

Temporarily connect to the internet for installation:

This is the one exception to the isolation rule. FLARE-VM must download its tools from the internet. Temporarily connect the VM to the External switch (through pfSense) or directly if needed, run FLARE-VM, then disconnect and restore isolation.

In pfSense, create a temporary rule allowing this VM's IP outbound internet access. After installation, delete the rule.

Run FLARE-VM

Open PowerShell as Administrator in the VM:

# Set execution policy to allow the script
Set-ExecutionPolicy Unrestricted -Force

# Download the FLARE-VM installer
(New-Object System.Net.WebClient).DownloadFile(
  'https://raw.githubusercontent.com/mandiant/flare-vm/main/install.ps1',
  "$env:USERPROFILE\Downloads\install.ps1"
)

# Run the installer
cd $env:USERPROFILE\Downloads
.\install.ps1

A GUI installer will appear asking which tool packages to install. For a complete analysis environment, select all packages — or at minimum:

• FLARE (core tools)
• Debuggers (x64dbg, WinDbg)
• Disassemblers (Ghidra, optionally IDA Free)
• File format tools (PEStudio, DIE, HxD, PE-bear)
• Utilities (Sysinternals, network tools)

The installation takes 30–60 minutes. The VM will reboot multiple times. This is normal — let it complete without interrupting.

Post-Installation

After FLARE-VM completes:

1. Disconnect internet access — remove the temporary pfSense rule, verify the VM is back on the Internal-Lab switch only
2. Verify isolation — ping 8.8.8.8 should fail
3. Take the baseline snapshot — this is the snapshot you'll restore after every analysis session:
   • In Hyper-V → right-click the VM → Checkpoint
   • Name: FLARE-VM-Baseline-Clean

This baseline snapshot is precious. Keep it. Never overwrite it.

Tour of FLARE-VM Tools

FLARE-VM installs a large number of tools. Here is a complete reference of the most important ones — what each tool is, what it does, when you use it, and professional alternatives:

Sysinternals Suite

The Sysinternals tools are free utilities from Microsoft that give you deep visibility into Windows internals. They are the foundation of dynamic analysis on Windows.

Process Monitor (Procmon)

What it is: A real-time monitor that logs every filesystem, registry, network, and process/thread activity on the system.

What it does: When malware runs, Procmon captures every file it creates or modifies, every registry key it reads or writes, every network connection it attempts, every process it spawns. It's the most comprehensive behavioral logging tool available for Windows.

When you use it: Running before and during malware execution. You'll have millions of events — filtering is essential (covered in Part 7).

Professional alternative: Procmon is itself the professional standard. Enterprise tools like CrowdStrike Falcon or Carbon Black also capture similar data but are commercial products.

Process Explorer

What it is: An enhanced replacement for Windows Task Manager.

What it does: Shows every running process in a tree structure showing parent-child relationships. Shows DLLs loaded by each process. Highlights suspicious properties — processes with no icon, processes running from temp directories, processes with mismatched signatures. Can submit process hashes to VirusTotal with one click.

When you use it: Running during malware execution to observe process creation, injection, and unusual process behavior.

Autoruns

What it is: A comprehensive scanner for persistence mechanisms.

What it does: Shows every program configured to run automatically on Windows startup — registry run keys, scheduled tasks, services, browser extensions, driver entries, and dozens of other persistence locations. Before running malware, take an Autoruns baseline. After running malware, take another. The diff shows exactly how the malware established persistence.

When you use it: Before and after dynamic analysis to identify persistence mechanisms.

TCPView

What it is: A real-time view of all active TCP and UDP connections.

What it does: Shows every open network connection and listening port, with the process name responsible for each. When malware establishes a C2 connection, TCPView shows it immediately — destination IP, port, protocol, and which process owns the connection.

When you use it: During malware execution to observe network behavior in real time.

ProcDump

What it is: A command-line tool for creating process memory dumps.

What it does: Captures a snapshot of a process's memory at a specific moment. Useful for extracting decrypted payloads from packed malware — after the malware unpacks itself in memory, ProcDump captures that unpacked state.

When you use it: When a packed or encrypted sample needs to be dumped from memory after it has decoded itself.

File Analysis Tools

PEStudio (Free)

What it is: A static analysis tool specifically designed for Windows PE (Portable Executable) files.

What it does: Examines a PE file without executing it and presents a comprehensive overview: file hashes, entropy, imports, exports, strings, sections, resources, certificates, and VirusTotal results. It automatically flags suspicious indicators — imports commonly used by malware, high entropy sections suggesting packing, missing or invalid signatures.

When you use it: First tool you open when you receive a new sample. It gives you a 60-second overview of what you're dealing with.

Professional alternative: PEStudio itself is used by professionals. For more advanced PE analysis, analysts use PE-bear or write custom Python scripts with pefile.

Detect-It-Easy (DIE) (Free)

What it is: A file type identifier and packer detector.

What it does: Analyzes a file and determines its type (PE32, PE64, .NET, ELF, etc.) and whether it has been compressed or obfuscated with a known packer (UPX, MPRESS, Themida, etc.). It uses a database of signatures called "detectors" to identify these.

When you use it: Right after PEStudio. DIE tells you whether the sample is packed (which means static analysis will be limited and you'll need to unpack it first) and what tools were used to pack it.

Professional alternative: ExeinfoPE (free) is another packer detector. For advanced unpacking, analysts use x64dbg's built-in OEP finding plugins.

PE-bear (Free)

What it is: A PE file structure viewer.

What it does: Shows the raw structure of a PE file — DOS header, PE header, section table, import table, export table, resource directory. More detailed than PEStudio for raw structure examination. Lets you edit PE fields directly.

When you use it: When you need to examine the raw PE structure in detail, particularly for understanding how a packer has modified the file or when verifying PE fields that seem anomalous.

HxD (Free)

What it is: A hex editor.

What it does: Shows the raw bytes of any file in hexadecimal and ASCII. Lets you read, search, and modify any file at the byte level. Essential for examining file magic bytes (the first few bytes that identify file type), finding embedded files, and reading data that string tools miss.

When you use it: When you want to look at raw bytes, search for patterns, or verify file type against its actual content (not its extension).

Professional alternative: 010 Editor (paid, ~$100) is the professional standard hex editor with binary templates for dozens of file formats. ImHex (free) is a modern open source alternative with template support.

FLOSS (FLARE Obfuscated String Solver) (Free, Mandiant)

What it is: An advanced string extractor.

What it does: Extracts strings from binaries including strings that have been obfuscated through XOR encoding, stack-based construction, or other techniques that simple string-dump tools miss. It emulates code execution to decode strings that would only be visible at runtime.

When you use it: After basic static analysis with PEStudio. FLOSS often reveals C2 URLs, file paths, registry keys, and error messages that were hidden from simpler tools.

Professional alternative: FLOSS is itself widely used professionally. Manual analysis in IDA Pro can reveal all strings but requires more skill.

Reverse Engineering Tools

Ghidra (Free, NSA/US Government)

What it is: A software reverse engineering framework developed and released by the National Security Agency.

What it does: Disassembles and decompiles binary executables — takes the raw machine code of a program and converts it into assembly language (disassembly) and pseudo-C code (decompilation) that humans can read and analyze. Supports dozens of CPU architectures and file formats. Has scripting support in Java and Python. Actively developed with a strong community.

When you use it: For deep static analysis when you need to understand exactly what the malware code does — not just what APIs it calls, but the logic of its algorithms, its decryption routines, its evasion techniques.

Professional alternative: IDA Pro ($3,000–$15,000+) is the industry standard disassembler and has been for 30 years. It's more mature and has better third-party plugin support. Binary Ninja ($500/year) is a modern alternative with excellent API and scripting. For beginners, Ghidra is more than sufficient and the concepts transfer directly to IDA Pro.

x64dbg (Free)

What it is: An open source debugger for Windows applications.

What it does: Lets you execute a program one instruction at a time, inspect registers and memory at each step, set breakpoints to pause execution at specific locations, and examine the program's state at any point during execution. Debugging is often used to unpack malware — you let the malware unpack itself in a debugger, then dump the unpacked code.

When you use it: For dynamic code analysis — when static analysis in Ghidra shows encrypted or packed code, you use x64dbg to watch the malware decrypt itself in real time.

Professional alternative: WinDbg (free, Microsoft) is the professional standard Windows debugger, used extensively for kernel-level analysis and crash dump examination. OllyDbg (free) is a classic that predates x64dbg and is still referenced in older tutorials. IDA Pro also includes a debugger.

IDA Free (Free, limited version)

What it is: The free version of IDA Pro — the professional standard disassembler.

What it does: Same core functionality as IDA Pro but limited to specific file formats (x86/x64 PE) and without the decompiler. Still extremely useful for analysts who want exposure to the IDA interface.

When you use it: As a supplement to Ghidra if you want practice with the IDA interface for professional preparation.

Networking Tools

Wireshark (Free, open source)

What it is: The industry standard network packet capture and analysis tool.

What it does: Captures every packet on a network interface and lets you examine them in detail — HTTP requests, DNS queries, TLS handshakes, C2 check-ins, data exfiltration, everything. Has powerful filtering and protocol dissection for hundreds of protocols.

When you use it: During dynamic analysis to capture all network traffic from the malware VM for detailed examination.

Professional alternative: Wireshark is itself the professional standard. Enterprise tools like Zeek (free) and Suricata (free) are used for network-level detection rather than interactive analysis.

FakeNet-NG (Free, Mandiant)

What it is: A dynamic network analysis tool that simulates internet services.

What it does: Intercepts all network traffic from the analysis VM and responds to common protocol requests — DNS queries get a response, HTTP requests get a response, FTP connections get accepted. Malware believes it has internet access and reveals its network behavior (C2 URLs, data it's trying to exfiltrate, protocols it uses) without actually connecting to the real internet.

When you use it: As an alternative to INetSim (which runs on REMnux). FakeNet-NG runs natively on Windows inside the analysis VM, making it convenient for quick analysis. INetSim on REMnux is more comprehensive for detailed network simulation.

Other Useful Tools Installed by FLARE-VM

CFF Explorer — PE file editor with detailed structure views. Complementary to PE-bear.

ResourceHacker — Extracts and edits resources embedded in PE files (icons, dialog boxes, strings, manifests). Malware often embeds payloads as resources.

PEview — Lightweight PE structure viewer. Faster than PE-bear for quick header checks.

dnSpy (Free) — Debugger and decompiler specifically for .NET assemblies. Many modern malware samples are written in C# or VB.NET — dnSpy gives you near-source-code level visibility into .NET malware without needing Ghidra.

Professional alternative for .NET: dotPeek (free, JetBrains) and ILSpy (free, open source) are also .NET decompilers. Commercial: .NET Reflector (~$100).

pestudio — Already covered above, but worth noting it has a VirusTotal integration that lets you check hashes directly from the tool.

7-Zip — Archive manager. Malware is often delivered in password-protected ZIP files (the password is "infected" by convention in the malware research community — this prevents mail scanners from extracting the sample).

Python 3 — Scripting language. You'll write Python scripts to automate repetitive analysis tasks, decode obfuscated strings, and process large amounts of data. Essential for serious analysis work.

Creating the Baseline Snapshot

After FLARE-VM installation is complete and all configuration is done:

1. Disconnect from the internet (remove the External switch connection)
2. Verify the VM is connected only to Internal-Lab
3. Verify ping 8.8.8.8 fails
4. Verify all FLARE-VM tools are accessible in Start menu
5. Verify Windows Defender is still disabled (it may re-enable after reboot)
6. In Hyper-V Manager: right-click VM → Checkpoint
7. Name: FLARE-VM-Baseline-[DATE]

This snapshot is the foundation of all your analysis work. Every time you finish analyzing a sample, you restore to this state. The VM is always clean, always ready.

Verification Checklist

• Windows VM installed and logged in as local analyst account
• VM connected only to Internal-Lab switch
• Internet access disabled (ping 8.8.8.8 fails)
• Windows Defender disabled
• Windows Update disabled
• FLARE-VM installed successfully
• Sysinternals tools available (Process Monitor, Process Explorer, Autoruns, TCPView)
• PEStudio, Detect-It-Easy, PE-bear, HxD installed
• Ghidra installed and launches
• x64dbg installed
• Wireshark installed
• FakeNet-NG installed
• Baseline snapshot created and named with date

Previous: Part 1 — Building the Lab Next: Part 3 — Setting Up REMnux and INetSim