Whitelist and blacklist configuration explanation

Types of configurations for whitelisting and blacklisting

SearchSG allows the following types of configurations for crawling your website and indexing them into the search engine:

Config	Purpose	Config applicable for
Whitelisted URL pattern list (internal)	You can provide us with a list of patterns you would like to whitelist in your internal page. We will not crawl/index any URL patterns that are outside of this list. You can pass us both a whitelist and a blacklist. To see how both lists interact, see below. Internal URL refers to any webpages that are in the same domain or subdomain as the domain we are crawling. For example, `search.gov.sg/results` and `sub.search.gov.sg/test` are both internal URLs of `search.gov.sg`	Crawling or Indexing pages
Blacklisted URL pattern list (internal)	You can provide us with a list of patterns you would like to blacklist in your internal page. We will not crawl/index any URL patterns that are in this list. You can pass us both a whitelist and a blacklist. To see how both lists interact, see below. Internal URL refers to any webpages that are in the same domain or subdomain as the domain we are crawling. For example, `search.gov.sg/results` and `sub.search.gov.sg/test` are both internal URLs of `search.gov.sg`	Crawling or Indexing pages

Configuration	Explanation
whitelistedUrlPatternList: `https://.stats.search.gov.sg/`, `https://.about.search.gov.sg/` blacklistedUrlPatternList: `https://.stats.search.gov.sg/test/`	When crawling, other than the two subdomains of `search.gov.sg`: `stats` and `about`, ignore all other pages. When crawling `stats.search.gov.sg`, we will skip pages with patterns like `stats.search.gov.sg/test` or `stats.search.gov.sg/test/page1`. This is because `stats.search.gov.sg/test` have been blacklisted.

The following table shows some FAQs for different URL types:

Types of URL	When the URL is a link on a page we are scraping, do we add it to the crawl queue to be scrap?	Will the document scraped from the page be added to crawl result?	In event of failure, will the document scraped from the page be added to error result?	Will URL links on the scraped page be added to crawl queue?	Remarks
URLs to follow (after consulting whitelist blacklist & `robots.txt`)	Yes	Yes	Yes	Yes	The `www.` subdomain is treated as a regular subdomain.
URLs do not follow (after consulting whitelist blacklist & `robots.txt`)	No	No	No	No	We treat these documents as though they do not exist on the site.

The following shows the truth table for whitelist, blacklist and robots.txt:

Whitelist	Blacklist	Robots.txt	Final Result	Remarks
Any	Disallow	Any	Disallow	If blacklist is present & on the blacklist, disallow
Disallow	Any	Any	Disallow	If whitelist is present & not on the whitelist, disallow
Allow	Allow \| Null	Disallow	Allow	Whitelist has priority over `robots.txt`
Null	Any	Disallow	Disallow	In absence of whitelist, if either blacklist or `robots.txt` disallow, disallow