Choose your rules and download the finished robots.txt file.
Upload robots.txt to the root of your site so it opens at yoursite.com/robots.txt. Blocking a page in robots.txt stops crawling but does not guarantee it stays out of search results; use a noindex tag for that.
AI crawlers like GPTBot and ClaudeBot collect pages for AI training and search. Block them here if you don't want that.