📘 This article identifies Botify's default list of resources in LogAnalyzer and explains how to modify this list to meet your needs. LogAnalzyer, part of Botify's Analytics Suite, is available with all Botify plans.
Overview
LogAnalyzer Crawl reports provide visibility into how bots crawl your resources, which is important to protecting your crawl budget. You should minimize Google crawling resources that are unimportant for page rendering. You can use Botify’s default list of resources or define your own.
💡 Create a warning segment to identify when resources are crawled in the Overview and Resources reports.
Default Resource List
URLs with the extensions shown in the following segmentation are considered resources in Botify by default:
[segment:internal_static]
@true
or (
path *.7z
path *.ai
path *.avi
path *.bmp
path *.bz2
path *.css
path *.csv
path *.dat
path *.dmg
path *.doc
path *.docx
path *.eot
path *.exe
path *.flac
path *.flv
path *.gif
path *.gz
path *.ico
path *.ief
path *.iso
path *.jpe
path *.jpeg
path *.jpg
path *.js
path *.json
path *.less
path *.m1v
path *.m4a
path *.m4v
path *.map
path *.mka
path *.mkv
path *.mov
path *.mp
path *.mp2
path *.mp3
path *.mp4
path *.mpa
path *.mpe
path *.mpeg
path *.mpg
path *.odt
path *.ogg
path *.ogm
path *.ogv
path *.opus
path *.otf
path *.ott
path *.ovg
path *.pbm
path *.pdf
path *.pgm
path *.png
path *.pnm
path *.ppm
path *.pps
path *.ppt
path *.pptx
path *.ps
path *.psd
path *.qt
path *.rar
path *.ras
path *.rdf
path *.rgb
path *.rss
path *.svg
path *.swf
path *.tif
path *.tiff
path *.torrent
path *.tsv
path *.ttf
path *.txt
path *.vcf
path *.wav
path *.webm
path *.webp
path *.wmv
path *.woff
path *.woff2
path *.woff2
path *.xbm
path *.xlm
path *.xls
path *.xml
path *.xpdl
path *.xpm
path *.xwd
path *.zip
}
Modifying the Default Resource List
If the default list of extensions does not meet your needs, define a custom list of URLs instead.
To define a custom list of resources:
Navigate to the Segment Editor from the main navigation bar.
With the Segments Overview article as guidance, copy the example segment from the Default Resource List and paste it at the bottom of the Segments Rules Editor window.
Edit the list to include the file types you want Botify to consider as resources, and save the segmentation.
Filtering Resources
When you have a segment defined for resources, you can filter your LogAnalyzer URL Explorer reports to include or exclude resources or add a report column to determine which URLs are defined as resources: