Auditing 573 URLs in one pass: log analysis + REST API on WooCommerce

A typical agency audit on a 500+ URL site takes 2–4 weeks and costs like a small Tesla. Half of that timeline isn't work — it's chasing admin logins, waiting for Search Console access, requesting SFTP, and emailing back-and-forth with a developer who's “on holiday.”

Here's how we run the same audit in one day, no admin access. Real case: a B2B catalogue of machine tools and metalworking equipment, WordPress 6.9 + WooCommerce + Avada, 409 products, 148 pages, 6 vendors in marketplace mode. 573 URLs total.

Step 1: Application Passwords and Rank Math API

First ask the client: an Application Password for a technical user with Editor rights. Built into WordPress since 5.6 — a token for the REST API without admin login. One POST request, one minute on the client's side.

GET /wp-json/wp/v2/pages?per_page=100&_fields=id,slug,title,status,link
GET /wp-json/wp/v2/posts?per_page=100
GET /wp-json/wc/v3/products?per_page=100
GET /wp-json/rankmath/v1/meta?ids=1,2,3...

Auth: Basic with base64 of user:app_password. No keys in the repo, no SSH. Runs from any CI worker.

Rank Math (one of two main SEO plugins for WP, alongside Yoast) exposes 57 REST endpoints since 1.0.230 — meta title, meta description, robots, schema, canonical, redirects. The entire SEO audit happens without admin login and without HTML parsing.

Step 2: what we look for

The audit script (Python, ~600 lines) iterates URLs and writes to SQLite. Dozens of fields per URL. The findings we extract in one pass:

Multi-H1

Classic page-builder bug: Avada, Elementor, Divi all happily generate pages with two or three H1s. On this case — 26 of 148 pages. Fixed by editing post_content directly through REST, without going through the Avada UI (the Avada UI would break the template on save).

noindex regressions

The sneaky ones. Three URLs returned noindex,nofollow: /logout/, /user/, /account-old/. First two — fine, they're utility. But /account-old/ was the client's promo page for legacy accounts and was supposed to be index.

Fixed via Rank Math REST field rank_math_robots. .htaccess got a 302 to root for the UserMembership router so Googlebot wouldn't hit the login form.

Product alt attributes

On a WooCommerce catalogue with 400+ products, image alt is always a pain. Client had 8,255 images, alt filled on 21%. Through wp/v2/media we mass-update: each image gets alt by template {product} — {category} — {angle}, with angle inferred from EXIF if available.

Result: 100% coverage. Batched 50 requests at a time, ~12 minutes for 8,000 images.

Sitemap duplicates

Rank Math generates an attachment sitemap by default. On a WooCommerce catalogue it inflates to 5,000+ entries of media-attachment URLs. Google indexes attachment pages as empty duplicates and starts dragging product rankings down.

On this case — 12 sitemap files before cleanup, 5 after (pages, posts, products, categories, glossary). Attachment sitemaps disabled via filter in functions.php.

Step 3: log analysis

We ask the client for access.log from the last 30 days (Beget or whichever host). On a WooCommerce site that's typically 200–800 MB. Parsed with GoAccess + custom scripts. We look for:

• what Googlebot and Yandexbot actually crawl — which URLs, how often, with what response code;
• 5xx spikes — on which URLs, at which hours;
• cache invalidation — does Googlebot revisit the same page with the same ETag and get 200 instead of 304;
• internal 404s — are we linking from our own pages to URLs that don't exist.

From the field: the month-long cache mystery

On this project we spent a month fighting the same thing: meta-description edits saved to the DB (visible in SQL), but on the live site reverted to the old value within 5 minutes. We thought WP Super Cache — flushed it, no luck. We thought Cloudflare — disabled it, no luck.

The reveal came over SSH (client granted access in week 6): the Beget server held two WordPress installs in the same parent directory. One — actually working, in ~/cmpnn.ru/public_html/. The other — a legacy install in ~/public_html/ with a symlinked .htaccess route overriding the modern one.

The REST API connected to the legacy install (client DNS pointed there), the live site served from the new one. Every edit we made went into a dead DB the live site never read.

Fix: 30 minutes. Remove the symlink, port all accumulated edits to the live DB, verify the REST endpoint. But you can't find this without SSH — REST returns 200 and says “saved everything.”

Classic “the cache is not a cache” story. Which is why our month-one checklist includes “confirm we're writing to whatever the production site actually serves from.”

What we closed in month one

• Full SEO audit of 573 URLs with P0/P1/P2 — single script run, ~8 minutes;
• 26 multi-H1 pages fixed via direct post_content edits;
• 36 empty product descriptions filled across 4 templates;
• 19 meta descriptions on commercial landings (including the cache-mystery resolution);
• alt attributes from 21% to 100% — all 8,255 images;
• 2 duplicate redirects, 3 noindex regressions fixed;
• sitemap pruned from 12 to 5 files.

This is not “a +35% traffic case study” — growth needs 3–6 months and depends on many factors. This is the visible foundation, without which no growth is possible. After this kind of work the site stops actively suppressing itself, and only then does it make sense to talk semantics, content and clusters.