By "technical SEO" I mean SEO problems that require software development.<p>What's special about domains with 100,000+ pages (e.g. niche search engines and large e-commerce sites) is that most basic SEO advice is necessary but far from sufficient.<p>I'm on a project with 1,000,000 pages and only about 50,000 are indexed. My "crawl budget" is absolutely small, but relatively large. How do I make sure the "right" content gets indexed? How does Google even define "right"?<p>I seem to be writing a lot of code to answer basic question like "which of my 1,000,000 pages are indexed?" and "which pages should I be actively seeking to index and de-index?"<p>If you work on "technical SEO" for something like Stack Overflow or Amazon I'd love to learn from you.<p>And if these are the wrong questions to ask, I'd especially like to learn from you.
I'm curious to hear other people's takes on this, too.<p>How unique is the content on each page? What do the crawled and not indexed vs discovered not crawled ratios look like relative to indexed pages? What does linking to the detail pages look like (category pages, similar items)? How much authority do category pages have in terms of backlinks? Page speed issues-- in GSC, can you manage average download time <100ms (makes a huge difference)?<p>Noindexing may help-- if you just initially handle that based on page content uniqueness and/or presumed MSV potential for the page. Otherwise, based on your language, can you request a quota increase from the 200/day notifications for the Indexing API?<p>If you like reading about the topic, maybe head to BHW forums (very poor SNR, though) and read about what some of the more unscrupulous people are doing to index MM+ pages/mo and extract some of the less-risky strategies (e.g. don't abuse the news/job posting APIs to get spam content indexed).<p>As additional commentary, the last 2-3 months have been <i>chaos</i> in terms of algo updates. There was essentially a 30 day period where Googlebot all but disappeared for most sites, then the two HCUs and other random updates tossed in for good measure, stacked with desktop infinite scroll and the switch to mobile first indexing. I can't remember as many changes happening in a single quarter in over 15 years of being SEO or SEO adjacent.
Finally an SEO question on HN! While I haven’t worked on SO, I’ve worked on a medical equivalent site which started with 1500 of 2.5m pages indexed and when I was finished with my contract they had over 1.4m indexed.<p>There are a few factors that you need to adjust for: authority, quality, structure.<p>Authority is mostly out of your hands as a technical SEO scope but as more content gets indexed, more links and signals will come in. The real culprits are quality and structure. As others have said, low-quality pages need to be addressed. They can always be worked back in later but you’re better to be ruthless about it now and add them back later.<p>Profile pages, thin content pages, duplicates all need to go. No index them first, then block in robots eventually and doesn’t hurt to canonical them where applicable. If you jump straight to robots, they won’t pickup the noindex directive.<p>The structure, a lot of people think of folder structure as IA but really it’s linking structure. It’s important to know where you’re placing emphasis on pages by how you’re internal links are setup. This is also the best way to surface deep pages as you can link to them to bring Google deeper into your site.<p>Also you can try refreshing dates and seeing if that helps. Short term solution but works plenty well.<p>I know this is pretty general but it’s hard without seeing your issues specifically. If I can help you get on the right track, lmk. Gladly take a look. I run a small(free) community with some strong technical SEOs in it as well who like helping. Not sure how to connect but lmk if interested and we will figure it out.
A few questions that come to mind off the bat:<p>Do you have a sitemap.xml file? Does it include accurate last updated dates or suggest realistic crawl schedules (eg: only index static pages every month, but frequently updated pages every day)?<p>Have you run your site through an SEO checker (or at least some of the ignored pages) to make sure that they are accessible and not being blocked/ignored by a crawler? Any server responses or canonical links that may signal the wrong thing to a bot?<p>Do you have any deep links to your more overlooked pages on the homepage (or at least on your more popular pages)?<p>Do you have backlinks in the wild that point to the pages that you want to have crawled? Are they marked as nofollow? Do you post links to these pages on any social media accounts?
Hmm maybe you can structure your links such that the high priority pages you want indexed are getting 10x more link juice than the rest? This would at least signal to Google which pages you deem more important to you