Advance Setup (recommended but optional)

Complete the following 3 setups to optimise your website for search

Setup Robots.txt

SearchSG crawler uses your robots.txt file to find the location of sitemap.xml.

Our crawler also abides by the rules you set in robots.txt.

To see how we crawl based on robots.txt and the whitelist / blacklist you furnish us, see here.

Intro to robots.txt

• A robots.txt file tells search engine crawlers which pages or files the crawler can or can’t request from your site.

  • You control which files crawlers may access on your site with a robots.txt file.

• robots.txt is a plain text UTF-8 encoded file that follows the Robots Exclusion Standard and lives at the root of your site.

  • Eg. for site www.example.com, the robots.txt file lives at www.example.com/robots.txt.

• A robots.txt file consists of one or more rules. Each rule blocks or allows access for a given crawler to a specified file path in that website.

• Unless you specify otherwise in your robots.txt file, all files are implicitly allowed for crawling.

Example robots.txt

User-agent: SearchSG
Disallow: /protected/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xmlCopy to clipboardErrorCopied

Setup Sitemap.xml

SearchSG crawler will look for a sitemap.xml first to find the list of resources that you want to crawled to appear inside your search results.

Intro to Sitemaps

• Sitemaps tell crawlers which pages are important to your site.
• To allow SearchSG to locate your sitemap, specifying the path to your sitemap in your robots.txt file.

Example robots.txt ref to Sitemaps

Sitemap: https://example.com/my_sitemap.xmlCopy to clipboardErrorCopied

• Sitemap files must be UTF-8 encoded, and URLs escaped appropriately.
• Current supported format sitemap file:
  • XML

Using Sitemaps in SearchSG

• Resources listed in sitemaps will be prioritised, crawled and updated more frequently by SearchSG.
• Use sitemap to control the list of resources that you definitely want to see in your search results.

Setup Structured Data

SearchSG supports publishing of structured metadata using schema.org vocabulary and JSON+LD data format, inline with standards used by other major web crawlers such as Google.

By making available additional structured metadata in a JSON+LD format on your page, you can allow SearchSG’s web crawler (as well as other web crawlers) to collect and index data according to your customised metadata and support your customisation needs.

For more information, see Search Filter & Result Item Customisation here.