# Site crawl brief

Before scraping a website, build a machine map.

## Order
1. Fetch `/robots.txt` — respect Disallow
2. Fetch `/sitemap.xml` if present
3. Fetch `/llms.txt` if present (preferred for AI-oriented sites)
4. Fetch `/api/catalog.json` or similar machine endpoints when linked
5. Only then fetch individual HTML pages — with size and timeout caps

## For agentstackkit.com specifically
- Map: https://agentstackkit.com/llms.txt
- Catalog: https://agentstackkit.com/api/catalog.json
- Prefer those over scraping every HTML page

## Output
Produce a crawl plan: allowed paths, priority URLs, and pages to skip. Use `crawl-map` / `fetch-safe` tools when available.