Skip to content

Sites, schedules and staging logins

Crawl rules, scan schedules, deploy hooks and scanning sites behind a login.

Updated

Crawl rules

  • Start URL: where the crawl begins.
  • Sitemaps: /sitemap.xml and any Sitemap: lines in robots.txt are read automatically.
  • Include and exclude: path globs to focus on parts of the site.
  • robots.txt is respected, using the user agent RonguAccessWatch or *.

At most two pages of a site load at once.

Schedules

OptionWhen it runs
WeeklyThe default for every site
DailyWith the Daily scans add-on
Deploy hookAfter each release, from your CI
Scan nowWhenever you click it

To use a deploy hook, create one on the site page and call the URL from your pipeline with a POST request. Keep the URL secret.

Staging sites behind a login

Two options, which can be combined:

  1. HTTP basic auth: a username and password.
  2. A login script: simple steps such as goto, fill, click and waitFor. No code is run.

Passwords are stored encrypted.

Alerts

When a scan finds new critical or serious problems, owners and admins get an email, and a Slack message if you added a Slack webhook.