CeWL · Solvere Labs

CeWL (Custom Word List generator) is a Ruby spider that crawls a site you are authorized to test and prints a wordlist built from the words that actually appear on its pages. Reach for it when a stock list like rockyou stalls and you suspect the target’s passwords are themed around its own language: product names, project codenames, staff jargon, team mottos. A generic list misses those; a site-specific one does not.

Mental model: cewl [options] URL -w wordlist.txt. Everything else is one flag away.

New to custom wordlists? Core concepts

Spider then extract: CeWL follows links from the start URL, then collects the unique words on the pages it reaches
Depth (-d): how many link levels deep to crawl, default -d 2; more depth means more words but a longer, noisier crawl
Minimum length (-m): drops words shorter than N so the list is not full of the, and, of, default -m 3
Offsite (-o): whether the spider follows links to other domains; off by default, keep it off unless those domains are in scope
Emails (-e) and metadata (-a / --meta): bonus harvests, addresses and document authors, useful as usernames
Frequency (-c): prints a count per word so you can rank the strongest candidates
Output (-w): writes the list to a file instead of the screen; always use it for real runs
A wordlist is candidates, not passwords: CeWL gives guesses, a cracking tool turns guesses into hits

When to reach for CeWL (and when not)

Reach for it when a generic list keeps missing and the box hints at themed passwords: WordPress and CMS challenges, product-named or jargon-heavy sites, or authorized assessments where you want a target-specific list to test password policy. It doubles as light OSINT, pulling emails and document authors as candidate usernames.

Reach for something else when you have not tried a generic list yet (start with rockyou or SecLists), when you have no web target at all (CeWL only works against a site), or when you know the exact password format rather than its vocabulary (use crunch to generate from a pattern). CeWL builds lists and never tests them, so pair it with hydra, john, or hashcat.

Install, update, verify#

sudo apt install -y cewl                    # Debian / Ubuntu / Kali
sudo gem install cewl                       # any host with Ruby + gem
sudo apt install --only-upgrade -y cewl     # update (or: sudo gem update cewl)

cewl --help | head -1   # version banner (no --version flag)
command -v cewl         # confirm it is on PATH
cewl --help | less      # skim the option groups

To run the newest code or read the source: git clone https://github.com/digininja/CeWL.git && cd CeWL && bundle install, then invoke it as ruby cewl.rb from that folder (same options). Metadata harvesting needs exiftool (libimage-exiftool-perl on Debian). Prove the install and your scope in one shot by crawling a page you control: cewl http://TARGET_IP/ should print a tidy column of words with no error trace.

Flags you’ll actually use#

Skim these first; the cheat sheet below is these flags combined.

Flag	Does	Flag	Does
`-d N`	Crawl depth (default 2)	`-c`	Show a count per word (rank)
`-m N`	Min word length (default 3)	`-o`	Follow offsite links (scope!)
`-x N`	Max word length	`--lowercase`	Lowercase all output
`-w FILE`	Write the wordlist to a file	`--with-numbers`	Keep words containing digits
`-e`	Also harvest emails	`-g N`	Also emit N-word groups
`--email_file F`	Emails to their own file	`-u UA`	Set the User-Agent
`-a` / `--meta`	Include document metadata	`-H`	Add a header (e.g. `Cookie:`)
`--meta_file F`	Metadata to its own file	`--auth_type`	HTTP `basic` / `digest` auth
`-n`	Suppress wordlist (with `-e`/`-a`)	`--proxy_host`	Route through a proxy
`--exclude F`	Skip paths listed in a file	`-v`	Verbose crawl progress

Cheat sheet#

Every block starts simple and adds one flag at a time, so you can stop at the line that does the job.

# Build up a crawl: stop at the line you need
cewl http://TARGET_IP/                            # 1. words to the screen (depth 2, min 3)
cewl http://TARGET_IP/ -w wl.txt                  # 2. + save to a file
cewl -d 3 http://TARGET_IP/ -w wl.txt             # 3. + go one level deeper
cewl -d 3 -m 6 http://TARGET_IP/ -w wl.txt        # 4. + keep only longer, password-worthy words
cewl -d 3 -m 6 -x 16 http://TARGET_IP/ -w wl.txt  # 5. + cap the maximum length

# Rank and normalize
cewl -c http://TARGET_IP/ -w wl.txt               # word,count pairs (frequent = stronger)
cewl --lowercase http://TARGET_IP/                # lowercase, print to stdout
cewl --with-numbers http://TARGET_IP/ -w wl.txt   # keep tokens like "spring2024"
cewl --lowercase http://TARGET_IP/ | sort -u > clean.txt   # dedupe and sort

# Harvest usernames alongside words
cewl -e http://TARGET_IP/ -w wl.txt               # also print emails
cewl -e --email_file emails.txt http://TARGET_IP/ -w wl.txt   # emails to their own file
cewl -a http://TARGET_IP/ -w wl.txt               # include document metadata (authors)
cewl -a --meta_file authors.txt http://TARGET_IP/ -w wl.txt   # authors to their own file
cewl -n -e -a --email_file e.txt --meta_file a.txt http://TARGET_IP/   # identities only, no wordlist

# Scope and reach control
cewl -o http://TARGET_IP/ -w wl.txt               # allow offsite links (only if in scope)
cewl --allowed '/blog/.*' http://TARGET_IP/ -w wl.txt   # only follow matching paths
cewl --exclude skip.txt http://TARGET_IP/ -w wl.txt     # skip paths listed in a file
cewl -g 2 http://TARGET_IP/ -w wl.txt             # also emit two-word groups

# Identity, auth, proxy
cewl -u 'Mozilla/5.0' http://TARGET_IP/ -w wl.txt # set a realistic User-Agent
cewl -H 'Cookie: PHPSESSID=LAB_SESSION' http://TARGET_IP/app/ -w wl.txt   # crawl behind a login
cewl --auth_type basic --auth_user u --auth_pass p http://TARGET_IP/ -w wl.txt  # HTTP basic auth
cewl --proxy_host 127.0.0.1 --proxy_port 8080 http://TARGET_IP/ -w wl.txt # route through Burp

# Feed the crackers
john --wordlist=wl.txt --rules=Jumbo --stdout > wl-rules.txt   # mutate: Company2024, etc.
hydra -l USER -P wl-rules.txt TARGET_IP http-post-form \
  '/login:user=^USER^&pass=^PASS^:Invalid'                     # spray a web login
hashcat -m 0 hashes.txt wl-rules.txt                           # crack captured MD5 hashes

Command breakdowns#

Each block below explains one situation, the exact command, and what to expect back.

Crawl one authorized page and save the list#

cewl http://TARGET_IP/ -w wordlists/base.txt

Default depth 2, minimum length 3. With -w set the screen stays quiet and the unique words land in the file. The fastest first pass on a new target.

Go deeper and keep only password-worthy words#

cewl -d 3 -m 6 http://TARGET_IP/ -w wordlists/deep.txt

-d 3 follows one more link level; -m 6 drops short filler so the cracker is not wading through the and and. A tighter, more useful list, and cracking runs faster.

Rank words by how often they appear#

cewl -c http://TARGET_IP/ -w wordlists/counted.txt

Writes word,count pairs. High counts are the site’s signature vocabulary and the strongest password candidates. Sort on the second field to float them to the top.

Harvest emails as candidate usernames#

cewl -e --email_file notes/emails.txt http://TARGET_IP/ -w wordlists/words.txt

-e turns on email collection; --email_file keeps them out of the wordlist so your password and username lists stay separate. Treat the addresses as candidates, not confirmed users.

Pull document metadata authors#

cewl -a --meta_file notes/authors.txt -d 3 http://TARGET_IP/ -w wordlists/words.txt

-a (same flag as --meta) runs exiftool over linked PDFs and Office docs and extracts author names, often real usernames. Depth matters here, since documents usually sit a level or two in.

Get identities only, skip the wordlist#

cewl -n -e -a --email_file notes/emails.txt --meta_file notes/authors.txt http://TARGET_IP/

-n suppresses the wordlist, so the run yields only the email and metadata harvest. Handy when you already have a password list and just want usernames.

Normalize and de-duplicate the output#

cewl --lowercase http://TARGET_IP/ | sort -u > wordlists/clean.txt

With no -w, CeWL prints to stdout so you can pipe it. --lowercase folds case, sort -u removes duplicates: a clean list ready to feed a cracker.

Keep tokens that contain digits#

cewl --with-numbers http://TARGET_IP/ -w wordlists/mixed.txt

By default CeWL keeps letter-only tokens. --with-numbers also accepts strings like spring2024 or v3lab, exactly the kind of token people bury in a password.

cewl -d 2 -H 'Cookie: PHPSESSID=LAB_SESSION_VALUE' http://TARGET_IP/dashboard/ -w wordlists/auth.txt

CeWL has no dedicated cookie flag; pass one with -H / --header. This reaches pages that only render after login on a box you control. For a server-level password box use --auth_type instead.

Authenticate to an HTTP basic or digest realm#

cewl --auth_type basic --auth_user labuser --auth_pass labpass http://TARGET_IP/private/ -w wordlists/priv.txt

For server-level auth (the browser credential prompt), not form logins. Switch to --auth_type digest if the server issues a digest challenge.

Constrain the crawl to one section#

cewl --allowed '/blog/.*' -d 4 http://TARGET_IP/ -w wordlists/blog.txt

--allowed only follows links whose path matches the regex, so a deep crawl stays inside /blog/ instead of wandering the whole app. Pair it with --exclude file.txt to blacklist noisy paths.

Route the crawl through Burp or a proxy#

cewl --proxy_host 127.0.0.1 --proxy_port 8080 http://TARGET_IP/ -w wordlists/site.txt

Every request lands in the proxy history for inspection and replay, so you can confirm exactly which pages CeWL touched and what it sent.

One-shot: build, save, and timestamp#

cewl -d 3 -m 6 -e http://TARGET_IP/ \
  -w "wordlists/site-$(date +%Y%m%d-%H%M).txt" 2>&1 | tee notes/cewl.txt

Deeper crawl, longer words, emails on, written to a dated file, with the console echoed to a log. Reproducible and easy to diff across runs.

Reading the output#

A column of words is the normal result: unique tokens that are candidates, not confirmed passwords.
word,count (with -c) ranks by frequency; the high counts are the site’s signature terms and the best guesses.
An empty list means the crawl never left the start page, -m filtered everything, or the page needs a cookie to render.
A giant list means depth is too high or filler slipped through; raise -m, lower -d.
Email and metadata blocks are bonus intel; treat names as candidate usernames, not facts.
Raw words rarely crack modern passwords; plan to add mutation rules (numbers, capitals, symbols) with john or hashcat.

Troubleshooting#

Problem	Cause	Fix
`command not found`	Not installed / stale PATH	Reinstall, or run `ruby cewl.rb` from the source clone
Empty wordlist	Crawl stayed on one page, or `-m` too high	Raise `-d`, lower `-m`, recheck the URL
Crawl never finishes	Depth too high on a large site	Lower `-d`, or scope it with `--allowed`
`Connection refused`	Wrong port or host down	Confirm the `URL` and that the service is up
SSL/TLS error	Self-signed lab cert	Try the plain `http://` URL on the lab box
Only login pages, no words	Auth required	Pass `-H 'Cookie: ...'`, or `--auth_type` for a basic realm
No metadata authors	No linked docs, or exiftool missing	Confirm docs are linked; install `libimage-exiftool-perl`
Old flag rejected	Version drift	Check the version banner and `cewl --help`

Gotchas#

-o is a scope leak waiting to happen. Left on, the spider follows offsite links and starts crawling third-party domains you were never authorized to touch. Keep it off unless those domains are explicitly in scope.
CeWL builds lists, it never tests them. Raw crawled words almost never match modern passwords on their own; run them through john --rules or a hashcat rule before judging the list as weak.
Metadata depends on exiftool. If it is not installed, -a returns no authors, so an empty metadata result can mean a missing dependency rather than a docless site.
The default User-Agent is a fingerprint. CeWL announces itself, and that single-source, breadth-first spidering burst is trivial to spot on a monitored target. Set -u to blend in and cap -d to stay quiet.
There is no cookie flag. Old writeups that show --cookie will error on current CeWL; the session goes in -H 'Cookie: ...' instead.
-m counts characters, -d counts link levels. Deep crawl plus a tiny minimum is the classic way to produce a massive list full of two-letter filler.

Pairs with#

wpscan feeds the custom list straight into WordPress logins, and hydra sprays it against other web and service logins. Captured hashes go to john or hashcat, which also apply the mutation rules that turn raw words into real guesses. theharvester complements the harvest with more emails and names, while ffuf and gobuster fuzz for the content worth crawling in the first place. Reach for a generic list (SecLists, rockyou) before CeWL, and for crunch when you know the password pattern rather than the site’s vocabulary.