.Google's John Mueller answered an inquiry regarding why Google marks pages that are forbidden coming from crawling through robots.txt as well as why the it's risk-free to dismiss the similar Search Console files regarding those creeps.Bot Traffic To Question Specification URLs.The individual asking the inquiry documented that crawlers were actually making links to non-existent inquiry specification URLs (? q= xyz) to web pages along with noindex meta tags that are also blocked in robots.txt. What prompted the question is actually that Google is crawling the hyperlinks to those webpages, receiving shut out by robots.txt (without envisioning a noindex robotics meta tag) then receiving shown up in Google.com Explore Console as "Indexed, though blocked by robots.txt.".The person talked to the observing inquiry:." However here is actually the significant question: why will Google.com mark pages when they can't even observe the material? What's the perk in that?".Google.com's John Mueller validated that if they can't creep the web page they can not see the noindex meta tag. He also makes an interesting reference of the web site: hunt operator, suggesting to ignore the end results due to the fact that the "typical" users will not see those outcomes.He composed:." Yes, you are actually correct: if our team can't creep the webpage, we can't find the noindex. That mentioned, if we can't crawl the webpages, then there's certainly not a lot for us to mark. So while you could find several of those pages along with a targeted website:- question, the normal user won't observe all of them, so I would not bother it. Noindex is also great (without robots.txt disallow), it just suggests the Links will wind up being actually crept (as well as wind up in the Explore Console document for crawled/not catalogued-- neither of these statuses lead to problems to the remainder of the site). The fundamental part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limitations being used the Web site: search evolved search driver for analysis reasons. Some of those causes is actually due to the fact that it is actually not hooked up to the routine hunt index, it's a different trait entirely.Google's John Mueller talked about the site search operator in 2021:." The quick response is that a site: question is certainly not indicated to become total, nor made use of for diagnostics purposes.A web site inquiry is a certain kind of hunt that limits the results to a particular internet site. It's basically merely the word web site, a bowel, and after that the website's domain name.This query limits the end results to a particular website. It's not suggested to be a comprehensive collection of all the webpages from that internet site.".2. Noindex tag without making use of a robots.txt is fine for these kinds of situations where a bot is connecting to non-existent webpages that are actually obtaining found out through Googlebot.3. Links along with the noindex tag are going to create a "crawled/not catalogued" item in Browse Console which those will not possess a damaging result on the remainder of the internet site.Read through the concern and also respond to on LinkedIn:.Why will Google.com index pages when they can not even view the information?Included Image by Shutterstock/Krakenimages. com.