Skip to main content

Optimizing by Latest Crawl

Updated over a year ago

📘 This article explains how to apply PageWorkers optimizations based on the latest crawl by Botify's crawler. PageWorkers is part of Botify's Activation Suite, available as an option with all Botify plans.

Overview

When creating PageWorkers optimizations, you can choose to apply the optimizations to your entire website or a subset of pages based on rules or specific URLs you define. You can apply the optimization based on the most recent SiteCrawler crawl for Replace URL, Remove URL, and Nofollow URL optimizations.

How it Works

When you apply an optimization by the latest crawl, PageWorkers integrates up to 10,000 URLs from your most recent SiteCrawler crawl based on the type of optimization you are configuring. For example, a Replace optimization displays URLs that returned 301 and 302 HTTP status codes during the crawl with their ultimate redirect destination URLs or non-canonical URLs with their canonicals. As with results in URL Explorer, you can apply additional filters to reduce the scope of results in PageWorkers.

Configuring an Optimization by the Latest Crawl

To configure an optimization by the latest crawl:

  1. Create and configure a Replace, Remove, or Nofollow optimization.

  2. On the Configure step, select Crawled URLs.​
    ​

    pw_optbycrawl.jpg

  3. For Replace URL optimizations, select the 301 or 302 URLs or Non-canonical URLs option.

  4. Optionally select metric filters to reduce the number of results. The results are displayed in the text box below.

  5. Choose one of the following actions to take when the source URLs are requested based on the type of optimization:

    • Add or do not add a 301/302 redirect to the ultimate redirect destination. Refer to Configuring Replace URL Optimizations for more information on these options.

    • Add/do not add a Noindex tag.
      ​

      pw_optbycrawl2.jpg

  6. On the Deploy step, click Save as draft. The optimization begins building, which may take several minutes, and the status is updated on the Optimizations Details page. When the build is complete, the Build in Progress button updates to Deploy. PageWorkers checks for completion every ten seconds; you do not need to reload the page to get the updated status.
    ​

    pw_buildinprogress.jpg

  7. Click Deploy to production to make the optimization live.
    ​

    pw_deploynew.jpg

Optimizations configured by the latest crawl are not updated after subsequent crawls; the URLs optimized at the time of configuration are static.

Did this answer your question?