Add rel="noindex" links to bw_navi output to reduce search engine requests and cache size

[bobbingwide]

I recently noticed that on seriouslybonkers.com the cache built by SiteGround's Speed Optimizer was approx 10GB.
Looking at the daily trace summary report I could see that Search Engines were submitting loads of requests with bwscidn=x query paramaters. The search engines were blindly following the pagination links produced by bw_navi()
for the following [bw_list] shortcode in the sequentially-biased.html template part.

<!-- wp:shortcode -->
[bw_list post_type=bigram posts_per_page=16 orderby=date order=DESC exclude=0]
<!-- /wp:shortcode -->

The posts_per_page attribute causes bw_navi to add shortcode pagination. When accompanied with the oik-ajax plugin the pagination is performed in the browser. It's pointless for the search engine to follow one of these navigation links as it basically produces the same result as the initial request.

If we add rel="nofollow" to the links then, hopefully.

1. Search engines won't waste the site's time
2. The cache won't become overloaded with requests no one will ever sensibly make.

Looking at today's summary report
Total requests: 17580
Requests with bwscidn=x format query params: 16375
Leaving: 1205 more normal requests

Search engines were responsible for:

• Babbar: 3012
• SemRush: 6015
• GoogleBot: 1768
• Ahrefs: 3943
• Bingbot: 821
• Facebook: 58

Total: 15559

Note: I didn't count Pinterest, Yandex and other sites
... but you get the gist.

[bobbingwide]

It's easy to change the links

In bw_navi_paginate_links() add a call to modify the generated $links

 $links = paginate_links( $args );
 $links = bw_navi_add_rel_nofollow( $links);

where the new function adds the rel="nofollow" attribute to each <a> tag,

function bw_navi_add_rel_nofollow( $links ) {
      $links = str_replace( '<a class', '<a rel="nofollow" class', $links );
     return $links;
}

I just need to decide if we can do this for all cases, or only when oik-ajax is activated.

[bobbingwide]

Should it be "nofollow" or "noindex"? I think it's the latter.

I'm going to test with "noindex". I may need to add some logic to make it optional in which case there'll be an attribute to set rel as nofollow, noindex or both.

I've uploaded the latest version of shortcodes\oik-navi.php to seriouslybonkers.com.
We'll see how long it takes for the search engines to notice.
And also how large the cache gets.

[bobbingwide]

We'll see how long it takes for the search engines to notice.

The following are still requesting stuff with different values of bwscid1

• facebookexternalhit
• ahrefs
• pinterest
• semrush
• GoogleBot
• AppleBot
• BingBot

Babbar didn't index the site on the 28th or 29th Oct.

And also how large the cache gets.

Cache is currently 4GB, which is a lot smaller. But there are several versions for each bigram.