SearchSG’s web crawlers visits and crawls through your domain to visit each page at least once per day during the day, this is to ensure that your search results are kept up-to-date.

Because of the low visits volume generated on your site, you do not need to worry about your website being crawled.

How do i identify SearchSG’s web crawler visiting my web page?

Robots.txt

User-Agent in robots.txt: SearchSG

Example:

User-Agent: SearchSG
Disallow:

User-Agent in HTTP Requests

via Crawl (Web Data source)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36; SearchSG

via API (API Data source)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36; SearchSGApiScraper

What pages are crawled and indexed for my website when using Web Data source?

By default, SearchSG crawls and indexes content within your website domain to power semantic search and AI-generated answers.

Primary Crawling Sources
SearchSG will:
- Crawl and index all URLs published in your sitemap.xml
- Discover and crawl additional pages within:
  - Your root domain - e.g. tech.gov.sg & www.tech.gov.sg
  - Your subdomains - e.g. info.standards.tech.gov.sg
This ensures comprehensive coverage of publicly accessible website content.
Automatically Indexed External Content
SearchSG also indexes commonly linked government content from your webpages that may not sit within your domain.
These include:
- FormSG forms hosted on form.gov.sg
- Documents/files hosted on:
  - isomer-user-content.by.gov.sg
  - file.go.gov.sg
  - files.app.optical.gov.sg
  - assets.app.optical.gov.sg

This allows search results to include relevant forms and documents even when they are hosted on different government platforms.

Customising What Gets Indexed

If you need to control what is included or excluded from search, you can configure:

Whitelist rules (explicitly include listed URLs only)
Blacklist rules (explicitly exclude listed URLs)

Configuration is completed within Admin Portal under Index Library > Select your Index > Data Sources > Select Web Data source to edit > Update whitelist/blacklist.

Example: To exclude all files hosted on file.go.gov.sg from search results, add the following blacklist rule:

https://file.go.gov.sg/*

What types of content on my website is being indexed for search?

We index the following content from your site:

Content Type	Indexing Scope
Webpages	All text content from pages in all domains and subdomains of your specified web domain
PDF, DOC, DOCX files	All text content extracted from document
Other files	Title and description only
Image files	Not indexed

This is how we determine what to index on your site:

What We Look At	Content Types We Index	Content Types We Don't Index
The HTTP response header "Content-Type"	Examples include: `text/html` `application/pdf` `application/msword` (DOC) ... and others	Any content where the HTTP response header 'Content-Type' starts with `image/`, such as: `image/jpeg` `image/gif` `image/png`
The path and domain of the URL	https://example.gov.sg/searchresults https://research.gov.sg	URLs with 'search' as an independent prefix in the domain or path (i.e., after a '/' and not part of another word): https://search.life.gov.sg https://search-gov.sg https://gov.sg/search-results/page-1

I followed the setup instructions but some of my webpages are still not appearing in the search results

We monitor our crawl results regularly to ensure that the number of results remains relatively stable and reduce the impact that transient errors may have on your page’s crawl results.

If certain pages in your site are not appearing in the search results, check your site’s robots.txt to ensure the content is allowed to be crawled and add the link to your sitemap.xml for prioritised crawling.

You can find out more in the section on how SearchSG uses robots.txt and sitemap.xml to crawl your website.

If your content is still not appearing even though you have allowed it to be crawled, our crawler might have run into some technical difficulties while processing your site.

Please reach out to our dedicated support team by submitting a service request.

Search Results Configurations

Why does my Website Search Bar redirect to an empty embedded search results page with only recommended links instead of search results of my query?

Your website’s security process or CMS (Content Management System) might have removed important URL query parameters needed by SearchSG’s application during page redirection.

Check that your website setup allows loading of the search results page with all the query parameters added by our Website Search Bar. If you observe a HTTP 301 redirection of results page to the same page but without the required query params, your website might require whitelisting of URL query parameters used.

If your website / CMS requires the whitelisting of query parameters, please whitelist required the query parameters in the table below:

Query params to whitelist	Usage for SearchSG
cid	clientId
q	query
page	page number
scope	search scope
sort	search sort
filter	search filter
sortList	search sort list
filterList	search filter list
contentType	search content filter
ac	autocomplete used (analytics)
sid	site Id (analytics)
duid	domain user Id (analytics)
dsid	domain session Id (analytics)
rurl	referrer (analytics)

My page is hosted on Isomer. How can I onboard SearchSG?

If you are still on Isomer Classic, you will have to migrate to Isomer Next by December 2026 to be able to use SearchSG. You do not need to separately create resources in SearchSG Admin Portal. For queries related to Isomer, you can reach out to support@isomer.gov.sg

Contact us to request for support

For any further assistance, please submit a service request form here.

Search APIs (Beta)

Troubleshooting

Crawling & Index

How often do SearchSG’s web crawler visit website?