Fetches /robots.txt and /sitemap.xml from a target, parses every Allow/Disallow rule, and flags interesting paths (admin/, backup/, api/, secret/, …).
Do you accept the TOS?