📘 This use case demonstrates modeling the trade-off between listing URLs removed and product pages orphaned, against your own crawl, internal link graph, and logs.
Best for: Technical SEO teams on large paginated sites · Uses: crawl, internal link graph, and logs, via your AI assistant
Overview
Paginated product listing pages (PLPs) generate large numbers of URLs, and capping pagination at a fixed depth is a common way to reduce crawl waste. The risk is orphaning product detail pages (PDPs) whose only internal link comes from a deep listing page. Botify MCP can model that trade-off against your own crawl, internal link graph, and logs, and show what each candidate cut-off actually costs.
Prerequisites
Botify MCP connected to your AI assistant, such as Claude or ChatGPT. See Connecting to the Botify MCP for setup steps.
A completed crawl with full internal linking data for the site.
LogAnalyzer ingesting logs, if you also want to see how deep search engine crawlers actually go.
Pagination deep enough to be worth analyzing.
Description of the problem
Choosing a pagination cutoff is usually a judgment call, because the data needed to make it properly sits in three different parts of the data model: the crawl, the internal link graph, and the server logs. Without joining them, you cannot tell which product pages depend on a deep listing for their only inlink, so any cutoff is a guess about how many pages you are about to orphan.
Application
Ask for both halves of the question in one request:
I'd like to check if I can close paginated PLPs on [organization]/[project]
- Can you visualize the percentage of pages crawled by Google, by number of active filters and page number?
- At what ideal depth should pagination stop to minimize the number of orphaned product detail pages (PDPs)?
Why this prompt works:Asking for both halves (crawl coverage by depth, and the orphaning model) in one request lets you see the trade-off in one pass, instead of running the coverage check and the orphaning model as two separate questions that are hard to line up afterward.
Naming the specific organization and project, rather than asking "how deep should pagination go" in general, is what lets the model use your own crawl, internal link graph, and logs rather than a general rule of thumb.
Review the crawl coverage by filter count and page number to see how deep search engine crawlers actually go. Depending on your AI client, this returns either a chart or a table.
Read the orphaning model by its marginal column, which shows what each additional level of depth buys you.
Validate the shortlist against merchandising and inventory before changing anything. Cutting pagination is a production change, and the model cannot see the commercial reasons why a deep listing matters.
Follow-up prompts
"Show me which specific PDPs would be orphaned at page [N]." — turns the aggregate count into a list merchandising can review.
"Re-run this after the next crawl." — checks whether the curve has shifted since the catalog changed.
Make it repeatable
Re-run the model after a meaningful catalog change or a new crawl, rather than treating one cutoff as permanent. The trade-off below is only as current as the crawl it's modeled against.
Once a cutoff ships, use Checking Whether a Ranking or Traffic Shift Lines Up With a Change to read what happened either side of it.
Result
Modeled against the internal link graph of a single crawl, roughly 500,000 compliant PDPs split into two groups. About 95% of them have at least one inlink from a non-paginated source and are safe at any cutoff. The exposure is the remaining 5%, reachable only through pagination. Cutting at page N orphans those whose shallowest paginated inlink sits deeper than N.
Cut at page | PDPs orphaned | % of PDPs | Listing URLs removed | Marginal PDPs rescued per 1,000 URLs kept |
2 | 13,500 | 2.70% | 154,000 | — |
5 | 7,000 | 1.40% | 119,000 | 186 |
10 | 4,000 | 0.80% | 92,500 | 113 |
15 | 2,900 | 0.58% | 77,500 | 73 |
20 | 1,850 | 0.37% | 67,500 | 105 |
25 | 1,500 | 0.30% | 59,500 | 44 |
50 | 850 | 0.17% | 37,000 | 29 |
100 | 350 | 0.07% | 18,500 | 27 |
The decision sits in the last column. Efficiency falls as the cut-off deepens: 186 pages rescued per 1,000 listing URLs kept at page 5, 113 at page 10, and 73 at page 15. It then rises to 105 at page 20 before dropping to 44 at page 25. Page 20 is where the curve turns. It removes 67,500 listing URLs while orphaning 1,850 product pages, 0.37% of the total, and every page kept beyond it buys less than half as much.
Read next
Related topics