Malware Analysis — Part 0: What Is Malware Analysis?

WARN

This series covers the analysis of malicious software in controlled, isolated environments. All techniques described are for educational and defensive security purposes only. The author accepts no responsibility for damage to hardware, software, data, or systems resulting from following this guide. Never analyze unknown files outside of a properly isolated lab environment as described in this series.

Before We Begin

This is Part 0 of a 14-part series on malware analysis. It contains no commands, no tool installations, and no technical steps. It is entirely conceptual — and that is intentional.

Every guide on malware analysis jumps straight to tools. Install this, run that, look at this output. The result is analysts who can follow steps but don't understand what they're looking for or why. This part exists to fix that. If you understand the concepts in this post deeply, every tool in every subsequent part will make immediate sense. If you skip it, you'll be following steps blindly.

Take your time with this one.

What Is Malware?

The word malware is a contraction of malicious software — any program or code designed to cause harm, gain unauthorized access, steal information, disrupt operations, or extort victims. The term covers an enormous range of software with very different goals:

Type	What it does
Virus	Attaches itself to legitimate files and spreads when those files are executed or shared. One of the oldest forms of malware.
Worm	Self-replicating malware that spreads across networks without needing to attach to a host file. Famous examples: WannaCry, Conficker.
Trojan	Disguises itself as legitimate software. Doesn't self-replicate but tricks users into running it. Often used to install other malware.
Ransomware	Encrypts victim files and demands payment for the decryption key. The most financially damaging category of malware today.
Spyware	Silently monitors the victim — keystrokes, screenshots, browser history, credentials — and exfiltrates the data to an attacker.
Rootkit	Hides itself and other malware deep in the operating system, often at the kernel level. Notoriously difficult to detect and remove.
Bootkit	Like a rootkit but infects the boot process itself — active before the OS even loads.
Backdoor	Creates a persistent, hidden access point into a system for the attacker to return to at will.
RAT (Remote Access Trojan)	A sophisticated backdoor with full remote control capabilities — file management, webcam, keylogger, shell access.
Dropper / Loader	A lightweight piece of malware whose only job is to download and install the real payload. Often used to evade antivirus.
Infostealer	Specifically targets credentials, browser data, cryptocurrency wallets, and sensitive files.
Botnet agent	Turns the infected machine into a bot controlled by a command-and-control (C2) server as part of a larger network of compromised machines.
Adware	Displays unwanted advertisements. Lower harm but often bundles spyware.
Fileless malware	Lives entirely in memory — never writes to disk, making it extremely difficult to detect with traditional antivirus.

Modern malware rarely fits neatly into one category. A real-world sample might be a loader that downloads a RAT which also functions as a keylogger with rootkit capabilities for persistence. Understanding these categories helps you recognize what a piece of malware is trying to accomplish when you analyze it.

What Is Malware Analysis?

Malware analysis is the process of examining a piece of malicious software to understand what it does, how it works, who wrote it, and what damage it has caused or could cause.

There are two fundamentally different ways to analyze malware:

Static Analysis

Static analysis means examining the malware without executing it. You look at the file itself — its structure, its contents, the code inside it — and draw conclusions about its behavior from what you observe.

Advantages:

Safe — you can't be infected by a file you don't run
Can examine malware that targets a different operating system or architecture than your analysis machine
Works even on samples that are broken or incomplete

Limitations:

Modern malware is almost always obfuscated, packed, or encrypted to defeat static analysis
You can see what the code says but not always what it actually does when running in a real environment
Requires significant expertise to interpret raw assembly code

Dynamic Analysis

Dynamic analysis means executing the malware in a controlled, isolated environment and observing its behavior in real time — what files it creates, what registry keys it modifies, what network connections it makes, what processes it spawns.

Advantages:

Shows you what the malware actually does, not just what the code says
Defeats many obfuscation techniques because the malware must eventually decode itself to run
Accessible to analysts at all skill levels

Limitations:

The malware might detect it's being analyzed and behave differently
Some behaviors only trigger under specific conditions — a certain date, a specific system configuration, a command from a C2 server
You must have a properly isolated environment or risk infecting real systems

The Relationship Between the Two

In practice, professional analysts use both approaches together, in a specific order. Static analysis first — gather as much information as possible without running anything. Then dynamic analysis — execute in a controlled environment and observe. Then back to static if needed — using insights from dynamic analysis to guide deeper code examination.

This series follows that order exactly.

Who Does This Work?

Malware analysis is performed by several different types of professionals, each with a slightly different focus:

Malware Analyst / Reverse Engineer The core role. Analyzes samples in depth, writes technical reports, and often develops detection signatures. This is the role this series most directly prepares you for.

Threat Intelligence Analyst Analyzes malware as part of understanding attacker campaigns, tactics, and motivations. Focuses on connecting technical indicators to threat actor groups and reporting on emerging threats.

Incident Responder Arrives after an organization has been compromised. Analyzes malware found during the investigation to understand what happened, what was stolen, and how to clean up. Needs malware analysis skills but applies them under time pressure in a real breach scenario.

SOC Analyst (Tier 2/3) Escalates alerts that may involve malicious files. Higher-tier SOC analysts need to be able to perform basic triage and initial analysis before escalating further or closing an alert.

Vulnerability Researcher Analyzes software — not necessarily malware — to find exploits. The skill overlap with malware analysis is significant, particularly in reverse engineering.

Detection Engineer Uses malware analysis findings to write detection rules (YARA, Sigma, Snort/Suricata rules) that help security products identify similar threats.

You don't need to choose a specialization now. The skills in this series are foundational to all of them.

Why Does This Matter?

If antivirus software exists, why do humans need to analyze malware manually? This is a reasonable question, and the answer reveals something important about the security industry.

Antivirus is reactive, not proactive. Traditional antivirus works by comparing files against a database of known malicious signatures. If a piece of malware has never been seen before — a "zero-day" — antivirus won't detect it. Someone has to analyze that new malware first, build a signature, and push it to every antivirus product in the world. That someone is a malware analyst.

Modern malware is designed to evade detection. Professional cybercriminal groups and nation-state actors specifically craft malware to bypass antivirus, sandbox systems, and automated analysis. Manual human analysis is often the only way to understand these samples fully.

Automated tools have real limits. Sandboxes and automated analysis platforms are enormously useful, but sophisticated malware can detect when it's being analyzed and go dormant. A human analyst can recognize these evasion techniques and work around them.

Understanding = better defense. If you understand exactly how a piece of ransomware encrypts files, you can design better defenses. If you understand how a loader downloads its payload, you can detect that pattern across your network. Analysis drives defense.

The Analyst's Mindset

Technical skills matter, but the most important thing you bring to malware analysis is mindset. Here's what separates good analysts from people who can just run tools:

Curiosity over certainty. You will rarely understand a sample completely. Good analysts are comfortable with partial knowledge and build their understanding incrementally, following evidence wherever it leads.

Methodical over impulsive. The temptation is always to run the sample immediately and see what happens. Disciplined analysts do static analysis first, gather as much context as possible, then execute with a prepared monitoring stack. Rushing leads to missed artifacts and inaccurate conclusions.

Skepticism about your tools. Every tool has blind spots. A sandbox that shows "no network activity" might have been fooled by an evasion technique. A decompiler that shows clean code might be misrepresenting obfuscated logic. Always cross-reference findings across multiple tools.

Documentation as you go. Malware analysis produces enormous amounts of data. Analysts who don't document their findings as they work inevitably lose important context. Your notes are your memory.

Patience. Some samples take hours. Some take days. Some are never fully understood. This is not a field for people who need immediate answers.

What You Will Need to Know

This series assumes you are a complete beginner to malware analysis but not a complete beginner to computing. Specifically, you should be comfortable with:

Windows basics — file system, registry, task manager, installing software
Basic networking — what an IP address is, what DNS does, what HTTP/HTTPS means, what a firewall does
Virtual machines — what they are conceptually, even if you've never set one up (we'll cover setup in detail)

You do not need to know:

Programming (helpful but not required for early parts)
Assembly language (we'll teach what you need in Parts 10 and 11)
Advanced networking (we'll explain as needed)
Any prior security experience

If you're coming from the IT career roadmap on this site, CompTIA Security+ level knowledge is more than enough to start. If you've built the certification homelab, your Proxmox and networking experience will directly apply.

A Note on Legality and Ethics

Malware analysis occupies a unique legal and ethical space. Some important principles:

Possession of malware samples for research is legal in most jurisdictions when done for defensive security research, but laws vary by country. Always research the laws in your jurisdiction before downloading samples.

Never run malware outside of an isolated environment. Not on your host machine. Not on a VM that shares a network with your home devices. Not "just quickly" to see what happens. One mistake can compromise your entire network, destroy data, or result in your machine joining a botnet.

Never use these skills against systems you don't own or have explicit written permission to test. The skills in this series are powerful. The ethical and legal boundary is clear: your own systems, your own lab.

Sharing samples responsibly. If you discover malware in the wild or develop signatures, share them through appropriate channels (VirusTotal, MalwareBazaar, your organization's security team) rather than distributing raw samples publicly.

The Tools You'll Encounter

This series uses a mix of free/open-source tools and mentions professional paid alternatives where they exist. Here's a high-level preview organized by phase:

Lab environment:

Hyper-V (free, built into Windows 11 Pro)
pfSense / OPNsense (free, open source)
REMnux (free, open source)
FLARE-VM (free, open source — from Mandiant/Google)

Static analysis:

PEStudio (free) — PE file examination
Detect-It-Easy (free) — file type and packer identification
PE-bear (free) — PE structure viewer
HxD (free) — hex editor
FLOSS (free) — advanced string extraction
Ghidra (free, NSA) — disassembler and decompiler
IDA Pro (paid ~$3,000+) — industry standard disassembler
Binary Ninja (paid ~$500/year) — modern alternative to IDA

Dynamic analysis:

Process Monitor (free, Sysinternals/Microsoft)
Process Explorer (free, Sysinternals/Microsoft)
Autoruns (free, Sysinternals/Microsoft)
TCPView (free, Sysinternals/Microsoft)
Wireshark (free, industry standard)
FakeNet-NG (free, Mandiant/Google)
x64dbg (free) — debugger
WinDbg (free, Microsoft) — professional-grade debugger
OllyDbg (free, legacy) — classic debugger, still used

Memory forensics:

Volatility 3 (free, open source)
Magnet AXIOM (paid) — commercial forensics platform

Automated sandboxes:

ANY.RUN (free tier available) — interactive sandbox
Hybrid Analysis (free) — automated sandbox
Triage by Hatching (free tier) — fast automated sandbox
Cuckoo/CAPE (free, self-hosted) — open source automated sandbox

How This Series Is Organized

Each part of this series builds directly on the previous one. Here's the full roadmap:

Part	Topic
Part 0	What Is Malware Analysis? (this post)
Part 1	Building the Lab — Hyper-V, pfSense, and Network Architecture
Part 2	Setting Up the Windows Analysis VM with FLARE-VM
Part 3	Setting Up REMnux and INetSim
Part 4	Safe Sample Acquisition, Triage, and Permissions
Part 5	Static Analysis Part 1 — File Identification and PE Structure
Part 6	Static Analysis Part 2 — Strings, YARA, and IOC Extraction
Part 7	Dynamic Analysis Part 1 — Behavioral Monitoring
Part 8	Dynamic Analysis Part 2 — Network Traffic and Sandboxes
Part 9	Memory Forensics with Volatility
Part 10	Introduction to Reverse Engineering
Part 11	Reverse Engineering with Ghidra — Deep Dive
Part 12	Writing a Malware Analysis Report
Part 13	Full Analysis Walkthrough — From Sample to Report

Before You Continue

Before moving to Part 1, spend some time reading published malware analysis reports from professional security teams. This will give you a concrete picture of what the end result of this process looks like:

You won't understand everything in those reports yet. That's fine — the point is to see where you're headed, not to understand it all today.

When you're ready: Part 1 — Building the Lab

This is Part 0 of the Malware Analysis series. Next: Part 1 — Building the Lab