← Back to Projects
Complete
aiself-hostingdockerwsl2nvidiagpulinuxhomelab

Self-Hosted AI Stack — Part 2: Environment Setup

Part 2 of the self-hosted AI stack series. Get a dedicated WSL2 distro running on the drive of your choice, with Docker Engine and GPU passthrough confirmed working before installing a single AI service.

N
NOTE

Goal for this part: get a dedicated WSL2 Linux distro running on the drive of your choice, with Docker Engine and GPU passthrough confirmed working, before installing a single AI service.

W
WARN

If you already use WSL2 for other development work, don't reuse your existing distro for this stack. Create a separate, dedicated distro instead — it keeps a bad dependency, a filled disk, or a botched update in this project from ever touching your regular development environment. This series assumes a fresh distro named Ubuntu-24.04, kept completely separate from any distro you already use day-to-day.

Why WSL2 instead of a VM or dual-boot

WSL2 gives you a real Linux kernel with near-native performance and, critically, working GPU passthrough to NVIDIA GPUs — without needing to dual-boot or dedicate a physical machine to Linux. For a Windows-based build like this one, it's the path of least friction while still getting you a proper Linux environment for Docker.

Installing a dedicated distro

wsl --install Ubuntu-24.04

If you already have a default distro installed, this adds a second, independent one without touching it. Confirm both exist and are separate:

wsl --list --verbose

Moving the distro to a dedicated drive

By default, WSL2 distros live on your system drive. If you want this stack on a separate drive — recommended, since model checkpoints get large fast — export and re-import it:

wsl --shutdown
wsl --export Ubuntu-24.04 D:\wsl-backup\ubuntu-24.04.tar
wsl --unregister Ubuntu-24.04
wsl --import Ubuntu-24.04 D:\WSL\Ubuntu-24.04 D:\wsl-backup\ubuntu-24.04.tar
T
TIP

wsl --import drops you into the distro as root by default. Set your normal user as default so you're not accidentally running everything as root:

ubuntu2404 config --default-user your_username

The exact command name matches whatever launcher WSL registered for your distro — check wsl --list if ubuntu2404 doesn't resolve.

Installing Docker Engine inside WSL2

Skip Docker Desktop for this — installing Docker Engine directly inside the WSL2 distro gives you a lighter setup with fewer layers between your containers and the GPU. From inside your Ubuntu-24.04 distro:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

Log out and back into the distro (or run newgrp docker) for the group membership to take effect, then confirm:

docker run hello-world

GPU passthrough: the NVIDIA Container Toolkit

This is the piece that lets containers actually use your GPU instead of just seeing a CPU. Two things need to be true:

  1. Your Windows NVIDIA driver needs WSL2 GPU support — recent drivers include this by default, no separate Linux driver install needed inside WSL2
  2. The NVIDIA Container Toolkit needs to be installed inside your distro
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
W
WARN

Blackwell-generation GPUs (RTX 50-series) require CUDA 12.8 or newer to be recognized correctly. This matters later when choosing container image tags — Part 3's ComfyUI image is deliberately pinned to a cuda12.8 tag for exactly this reason. An older CUDA tag can silently fail to detect the GPU or fall back to CPU-only execution. If you're on an older GPU (Ampere, Ada Lovelace), CUDA 12.x in general is still recommended but the specific 12.8 pinning is less critical.

Verifying GPU passthrough end-to-end

docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi

If this prints your GPU's name, driver version, and memory — passthrough is working, and you're ready for Part 3.

W
WARN

If this command errors out, don't move on yet. Every service in this stack depends on GPU passthrough working correctly. Debug here before installing anything else.

Troubleshooting

nvidia-smi works on Windows but fails inside the container

Confirm your Windows NVIDIA driver version supports WSL2 GPU passthrough — recent drivers do by default. Also confirm you're not running an old cached Docker image:

docker pull nvidia/cuda:12.8.0-base-ubuntu24.04

Pull the image fresh and retry. If it still fails, check that the NVIDIA Container Toolkit installation completed without errors — re-run the toolkit install commands and look for any apt errors.

wsl --import distro won't start, or hangs

Confirm the target path (e.g. D:\WSL\Ubuntu-24.04) exists and that you have write permissions there. Also confirm wsl --shutdown fully stopped all running distros before the export step — an export mid-write can produce a corrupted tarball.

wsl --list --running

If anything is still listed, shut it down before exporting.

Docker commands require sudo every time

Your user wasn't added to the docker group, or the group change hasn't taken effect in the current shell session. Check with:

groups

If docker isn't listed, the usermod command didn't apply yet. Fully close and reopen your WSL2 terminal — not just newgrp docker — and check again. If it's still missing, re-run:

sudo usermod -aG docker $USER

Then close and reopen the terminal completely.

Series Navigation