.Google's John Mueller answered a question regarding why Google indexes webpages that are actually forbidden coming from crawling by robots.txt and why the it's risk-free to dismiss the similar Look Console reports concerning those crawls.Crawler Visitor Traffic To Question Parameter URLs.The individual talking to the inquiry documented that crawlers were creating hyperlinks to non-existent inquiry guideline URLs (? q= xyz) to webpages along with noindex meta tags that are also blocked in robots.txt. What triggered the question is that Google.com is crawling the links to those pages, obtaining blocked through robots.txt (without watching a noindex robots meta tag) after that getting reported in Google Search Console as "Indexed, though blocked by robots.txt.".The individual asked the adhering to question:." Yet here is actually the major inquiry: why will Google.com index webpages when they can't also view the material? What is actually the benefit because?".Google's John Mueller verified that if they can not crawl the web page they can't view the noindex meta tag. He likewise produces an interesting reference of the internet site: hunt driver, recommending to dismiss the results given that the "normal" users won't find those results.He composed:." Yes, you're proper: if our team can not creep the web page, our team can't view the noindex. That stated, if our experts can not crawl the pages, after that there's not a whole lot for us to index. So while you could observe a number of those webpages with a targeted web site:- inquiry, the normal consumer won't see them, so I definitely would not bother it. Noindex is additionally great (without robots.txt disallow), it only suggests the URLs will definitely find yourself being actually crept (and wind up in the Explore Console report for crawled/not indexed-- neither of these conditions cause problems to the rest of the site). The essential part is that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the constraints in using the Web site: search progressed hunt driver for diagnostic reasons. Among those main reasons is actually given that it's certainly not linked to the frequent search mark, it is actually a distinct thing altogether.Google.com's John Mueller commented on the website hunt driver in 2021:." The brief answer is actually that an internet site: inquiry is certainly not indicated to become complete, nor used for diagnostics objectives.An internet site concern is actually a specific type of search that confines the end results to a particular website. It is actually primarily just words site, a bowel, and then the internet site's domain name.This query confines the results to a certain site. It is actually not implied to be a complete compilation of all the web pages coming from that website.".2. Noindex tag without making use of a robots.txt is alright for these sort of situations where a robot is actually connecting to non-existent web pages that are actually receiving uncovered through Googlebot.3. Links with the noindex tag will certainly produce a "crawled/not recorded" entry in Browse Console and also those won't have a negative effect on the rest of the internet site.Read through the concern and also answer on LinkedIn:.Why would certainly Google.com index pages when they can't also observe the web content?Featured Photo by Shutterstock/Krakenimages. com.