Website Crawler

Make your website content searchable in minutes with the Sajari crawler, no programming required.

quickstart

Up and running in minutes

Get your website search up and running in minutes with the Sajari crawler. Simply enter your domain name and the crawler will do the rest.

By automatically extracting content from your websites, Sajari’s crawler removes the need for custom development, getting you started faster.

always up-to-date

Instant indexing

Make changes to your site available in an instant. Instead of waiting for your content to be crawled periodically, Sajari will discover and index content as soon as a new page is viewed or an old one is removed.

Automatic categorization

Leveraging your website’s metadata, your content will automatically be categorized allowing you to group results by context, topic, type, location, date, or whatever categories are relevant to your business.

Custom field extraction

Turn your websites into structured content by using custom tags to add more information to your pages. Gives your users even more flexibility to search and discover information.

PDF & MS Office document indexing

Surface information hidden in PDF or Microsoft Office documents to make them searchable. The crawler automatically indexes your documents and makes the content available in the search results.

Index debugging

Optimize your documents by understanding what exactly is being indexed.

site search

Cross-domain search

Create a unified search experience across multiple sites, no matter what the underlying technology may be. Aggregate information from different websites or bring together your website, documentation, blog, and community forum under a single search experience.

Optimized for your visitors

Search crawler rules

Tell the crawler which parts of your website are off limits to ensure that only the content you want searchable is accessible.

API

Site integration

Need even more flexibility? Sajari offers a rest API and SDKs in a number of languages to help you make all your content searchable, no matter where it lives and how it’s structured.