wget

wget is a non-interactive downloader: give it a URL and it writes the content to disk, resuming broken transfers and mirroring whole directory trees without a browser. Reach for it when the deliverable is a file on disk rather than one crafted request, whether you are pulling a wordlist onto a lab box, mirroring a small site for offline review, or fetching inside a shell where it is the only downloader present. The moment you care about the request itself (headers, methods, an API), hand off to curl.

Mental model: wget [options] URL saves a file. Everything else tunes where it lands and how far it recurses.

Install, update, verify#

sudo apt install -y wget                  # Debian / Ubuntu / Kali
sudo dnf install -y wget                  # Fedora / RHEL
brew install wget                         # macOS (Homebrew)
sudo apt install --only-upgrade -y wget   # update

wget --version | head -1   # version + TLS build (matters for HTTPS)
command -v wget            # confirm it is on PATH

wget ships on most Linux distros, so install is usually just a confirmation. On Windows, install it with winget install JernejSimoncic.Wget, choco install wget, or scoop install wget (Git Bash ships curl.exe, not wget). The main reason to update is newer TLS so HTTPS keeps working against modern servers.

Flags you’ll actually use#

Skim these first; the cheat sheet below is these flags combined.

Cheat sheet#

Every block starts simple and adds one flag at a time, so you can stop at the line that does the job.

# Build up a mirror: stop at the line you need
wget URL                                 # 1. one file, remote name
wget -r -np URL                          # 2. + recurse, never climb above this dir
wget -r -np -l 2 URL                     # 3. + cap depth at 2 levels
wget -r -np -l 2 -p -k URL               # 4. + page requisites, rewrite links = offline copy

# Download one file
wget -O out.bin URL                      # save to a chosen name
wget -P downloads/ URL                   # save into a folder, keep remote name
wget -c URL                              # resume a partial download
wget -nc URL                             # skip if the file already exists (no-clobber)
wget -N URL                              # re-download only if the server copy is newer
wget --content-disposition URL           # honor the server's suggested filename
wget -i urls.txt -P downloads/           # fetch every URL listed in a file

# Inspect without saving
wget --spider -S URL                     # does it exist? print headers, save nothing
wget -S -O /dev/null URL                 # status + headers, discard the body

# Recursive / mirror / filter
wget --mirror -np -p -k -P mirror/ URL   # offline-viewable copy of a small site
wget -r -np -A pdf,zip URL               # recurse, keep only .pdf and .zip
wget -r -np -R "*.js,*.css" URL          # recurse, drop matching files
wget -r -np -e robots=off URL            # ignore robots.txt (authorized mirrors only)

# Send data / auth / identity
wget --post-data='user=U&pass=P' URL     # POST a form body
wget --user=USER --password=PASS URL     # HTTP basic auth
wget --header="Authorization: Bearer TOKEN" URL   # bearer token
wget -U "Mozilla/5.0" URL                # set the User-Agent

# Cookies / session
wget --save-cookies jar.txt --keep-session-cookies \
  --post-data='user=U&pass=P' URL        # log in, save the session cookie
wget --load-cookies jar.txt URL          # replay the saved session

# Quiet / resilient / polite
wget -q -O out URL                       # silent on success (check $?)
wget -b URL                              # run in the background, log to wget-log
wget --tries=2 --timeout=10 URL          # fail fast on a dead host
wget --limit-rate=200k --wait=1 --random-wait URL   # cap speed, space out requests
wget --no-check-certificate https://TARGET_IP/      # lab self-signed TLS

Command breakdowns#

Each command does one thing and leaves you a file (or an answer) you can act on.

Download a file, keeping its name#

wget http://TARGET_IP/files/notes.txt

Lands notes.txt in the current folder. If it already exists, wget writes notes.txt.1; it never silently overwrites without -O. Add -nc to skip the fetch entirely when the file is already there.

Save to a chosen name or folder#

wget -O downloads/loot.bin "http://TARGET_IP/download?id=42"   # rename
wget -P downloads/ http://TARGET_IP/files/report.pdf          # keep name, choose dir

Use -O for ugly generated URLs; use -P to keep the original name but organize by target.

Resume an interrupted download#

wget -c http://TARGET_IP/files/big.iso

wget requests the missing byte range and continues, but only if the server honors Range. If it restarts at zero, the server does not support it.

Fetch a whole list of URLs#

wget -i urls.txt -P downloads/

One URL per line, each fetched in turn; failures are reported per line and do not stop the batch.

Check a file exists without downloading it#

wget --spider -S http://TARGET_IP/files/backup.zip

Prints the status line and headers, saves nothing. The fast way to confirm a path (and its Content-Length) before committing to the transfer.

See the server’s response headers#

wget -S -O /dev/null http://TARGET_IP/

Dumps the full HTTP/1.1 200 OK plus header block, then throws the body away: triage without cluttering the disk.

Mirror a small site for offline viewing#

wget --mirror --no-parent --page-requisites --convert-links \
  -P mirror/ http://DOMAIN/docs/

--page-requisites grabs CSS/JS/images; --convert-links rewrites them to open locally; --no-parent keeps it under /docs/. Open the result straight from disk.

Download a directory but nothing above it#

wget -r -np -l 2 http://TARGET_IP/files/

-r recurses, -np refuses to climb to the parent, -l 2 caps depth so one stray link does not walk the whole tree.

Grab only certain file types#

wget -r -np -A pdf,zip,conf http://TARGET_IP/share/

-A is an accept-list by extension (-R rejects). wget still crawls the tree but keeps only matches.

Ignore robots.txt while mirroring#

wget -r -np -e robots=off http://TARGET_IP/restricted/

By default wget obeys robots.txt and may fetch nothing; -e robots=off fetches pages a crawler rule would skip, on a site you own or are authorized to mirror.

Run a big download in the background#

wget -b http://TARGET_IP/files/huge.tar.gz

Detaches and logs progress to wget-log in the current dir. Run tail -f wget-log to watch it.

Spoof the User-Agent#

wget -U "Mozilla/5.0" -O downloads/page.html http://TARGET_IP/

Some servers block the default Wget/1.x agent; a browser string gets past a naive filter.

Authenticate with HTTP basic auth#

wget --user=USER --password=PASS -O out.txt http://TARGET_IP/private/file.txt

Returns the file on valid creds, 401 otherwise. --user/--password cover both HTTP and FTP; --http-user/--http-password scope the same creds to HTTP only.

Reuse a logged-in session via cookies#

wget --save-cookies cookies.txt --keep-session-cookies \
  --post-data='user=admin&pass=hunter2' http://TARGET_IP/login
wget --load-cookies cookies.txt http://TARGET_IP/dashboard

First call logs in and stores the session cookie; second call replays it. --keep-session-cookies is what preserves the login.

Fail fast on a dead host#

wget --tries=2 --timeout=10 -O out http://TARGET_IP/file

Gives up after two short attempts instead of hanging, which is essential inside scripted sweeps.

Stay quiet and low-noise#

wget --limit-rate=200k --wait=1 --random-wait -r -np -l 2 http://DOMAIN/docs/

Caps bandwidth and spaces requests to stay polite and avoid hammering a server you are authorized to mirror.

Reading the output#

HTTP request sent, awaiting response... 200 OK          # the status
Length: 1048576 (1.0M) [application/octet-stream]       # expected size + type
...
2026-05-24 12:00:00 (5.4 MB/s) - 'big.iso' saved [1048576/1048576]   # got/expected

A stalled progress bar means a slow or hung link (tune --timeout/--tries). A file smaller than the reported Length is a truncated transfer, so verify size and hash before trusting it.

Troubleshooting#

Gotchas#

• -O and -o are different flags. -O sets the output file; lowercase -o redirects wget’s log. Swap them and your download vanishes into a log path.
• -O does not play well with -c or -N. A fixed output name plus resume/timestamping can restart the transfer or error out. To resume, use -c with the default name (or -P).
• A saved 404 is still a file. wget writes the error page to disk; check the status line and byte count, do not assume the bytes are what you asked for.
• --mirror means infinite depth. Always pair it with --no-parent (and ideally -l N) or one stray link walks the whole site.
• The default agent says Wget/1.x. Some servers block it, so set -U to a real browser string if a fetch is rejected.
• --no-check-certificate accepts any cert. Fine on a lab box, dangerous baked into a script as a habit.

Pairs with#

curl -v takes over the moment you need to read headers or craft the request, which is where wget hands off. nmap finds the open web ports first, then gobuster and ffuf discover the paths worth pulling. sha256sum confirms a transfer finished intact. For large files, aria2c -x16 parallelizes the download and httrack has friendlier controls for a serious offline site copy, but wget stays the zero-setup default already on every box.