How to exclude bot from appearing on certain pages
Wanting to prevent your bot from appearing on certain pages of your website? You can use a targeting rule that allows you to exclude specific pages by their URL or by certain words in the URL. Here's how you can do that:
- 1
- Open the bot you would like to apply the targeting rule on.
- 2
- Go to Targeting at the top, scroll down till you reach the section to select a specific rule and choose Page URL.
- 3
-
You have five different conditions to choose from, what we are interested in is
- is not: Enter the full URL of the web page you would like to exclude(e.g.https://www.website.com/store/shoes/blue-shoes)
- doesn’t contain: Enter a part of the URL that you would like to exclude, if that part exists in any URL it will exclude it from appearing,(e.g. if we add“shoes” then any URL that has the word“shoes” in it will be excluded)
- 4
- Click on Publish on the top right to make the changes go live.
Note: If you are adding multiple pages to be excluded, make sure you choose to match All of these rules to exclude them all.
Note:
When we compare your URL to the conditions you added we take the URL and remove the last '/' (at the end of the URL) from it and then take the pattern you entered and also remove the last '/' if there is one. So if your URL is https://website.com/categories/, and you add a rule Page URL contains '/categories/ 'we will be using https://website.com/categories and /categories to make a match.
You can turn this feature OFF with our new 'Do not remove trailing slash' option. It makes the match literal.