Spidering

katana

# go install github.com/projectdiscovery/katana/cmd/katana@latest
cat hosts | katana -jc -kf all -nc -ef png,jpg,jpeg,css,gif,ttf,woff,woff2,svg,eot

Headless mode reaches the routes a static crawl never sees: SPA views, lazy-loaded chunks, XHR endpoints.

katana -u https://example.com \
  -hl -jc -kf all -d 5 -aff -fx -xhr \
  -ef png,jpg,jpeg,gif,svg,woff,woff2,ttf,eot,ico \
  -o katana.txt

-hl    headless
-jc    crawl JavaScript
-kf    known files: robots.txt, sitemap.xml
-aff   automatic form fill, gets past gated flows
-fx    crawl out-of-scope forms/JS for endpoints
-xhr   capture XHR requests, not just links
-d     depth

Authenticated crawl, most of the app is behind login:

katana -u https://example.com -H "Cookie: session=..." -hl -jc -d 5

Passive first

Someone already crawled it. Start there, it costs nothing and hits deleted content.

# https://github.com/lc/gau
gau --threads 10 example.com > gau.txt

# https://github.com/tomnomnom/waybackurls
echo example.com | waybackurls > wb.txt

# https://github.com/projectdiscovery/urlfinder
urlfinder -d example.com -all -silent

Then merge and dedupe by shape, not by string. Thousands of URLs collapse to a few hundred templates:

# https://github.com/s0md3v/uro
cat gau.txt wb.txt katana.txt | uro > urls.txt

# https://github.com/tomnomnom/anew
cat new.txt | anew urls.txt

Mine the output

# parameters worth testing
grep -oP '(?<=[?&])[a-zA-Z0-9_\[\]-]+(?==)' urls.txt | sort | uniq -c | sort -rn | head -50

# files that should not be there
grep -Ei '\.(bak|old|sql|zip|tar\.gz|log|env|json|yml|yaml|conf|pem|key|pdf|xls)' urls.txt

# juicy paths
grep -Ei '(admin|internal|debug|test|staging|api|graphql|swagger|actuator|\.git|backup)' urls.txt

# by extension, to see what the app is built on
grep -oE '\.[a-z0-9]{2,5}($|\?)' urls.txt | sort | uniq -c | sort -rn | head

# https://github.com/tomnomnom/unfurl
cat urls.txt | unfurl -u keys | sort -u        # every parameter name
cat urls.txt | unfurl -u domains | sort -u     # every host referenced

Robots, sitemaps, and the rest

/robots.txt          /sitemap.xml        /sitemap_index.xml
/.well-known/security.txt
/swagger.json  /openapi.json  /v2/api-docs  /swagger-ui.html
/graphql  (introspection)
/.git/config  /.svn/entries  /.DS_Store
/actuator  /actuator/env  /actuator/heapdump

robots.txt disallow entries are a map of what they did not want indexed.

Then

Javascript: the endpoints in the bundles
Param Discovery: the parameters no URL shows
Fuzzing: what nothing linked to

Find records

# https://github.com/projectdiscovery/dnsx
dnsx -retry 3 -a -aaaa -cname -ns -ptr -mx -soa -resp -silent -l subdomains.txt > dnsx_info.txt