Google Crawling and Indexing Tips
Once a website owner starts to get to a slightly more advanced stage of search engine optimization they are going to want to view some highly detailed and technical information regarding how their website is crawled. Crawl rate and the ability for search spiders to access certain aspects of a website can play a significant role in search engine optimization and marketing over time. Robot.txt files play a significant role in how website pages often times index, and if positioned improperly can really be disastrous to a website.
“Do you know how Google’s crawler, Googlebot, handles conflicting directives in your robots.txt file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot’s favorite song? The answers to these questions (except for the last one 
), along with lots of other information about controlling the crawling and indexing of your site, are now available on code.google.com:”
Code.Google.com is a source put out by Google where webmasters can turn to for information and resources surrounding the robot.txt files for their websites. I’m not going to sugar coat it, but if you are not a very technical person this Google website might not be your cup of tea. However, if you enjoy going through code and dissecting the technical aspects of your website then you will enjoy visiting this Google site.
“Now site owners have a comprehensive resource where they can learn about robots.txt files, robots meta tags, and X-Robots-Tag HTTP header directives.”
The internet consistently becomes a complicated area to market a business. The landscape is extremely dynamic and things often change overnight with little to no warning, which is why resources such as this continue to be very important in the website marketing community. Behind every great online marketing campaign comes a strategic technical component backing the whole process making sure that all of the i’s are dotted and t’s are crossed.
9 Responses to “Google Crawling and Indexing Tips”
SEO Journal is Written by
Brick Marketing Founder,
Nick Stamoulis:
Nick Stamoulis - SEO Bio.
Tel. - 877-295-0620
nick@brickmarketing.com
Connect with Nick Stamoulis
Brick Marketing - SEO Company
thanks for this informative article. I’m not an expert seo but I can achieve good results for my sites and my clients sites. at one point I’m very close to the top 10 for choice keywords but I cannot advance anymore. a bit frustrating. I’m sure little tips like these can make the difference between success and failure. I particularly like the last paragraph because a lot of non-technical people don’t understand the complexities involved in this business.
Hey great article, I didn’t know that stuff was available at code.google.com even though I’ve browsed there many times.
But now I’m going to be obsessed with figuring out Googlebot’s favorite song! *L*
You are correct about the technical aspect. I know enough to be dangerous with all of the information at code.google.com. Fortunately, technical assistance is not too far away. Thanks for making us aware of this resource.
Googlebot’s favorite song is Itsy Bitsy Spider.
Thanks for this information today. Funnily enough (well, not actually) I’ve currently got a crawl error which I have tried to get resolved through Blogger, to no avail so far. I will check out Code.Google.com right away to see if I can understand it!!
Hi Nick,
I read all your SEO tips delivered to my in box regularly. It’s guys like you who help a small site owner like me to stand the might of Google and other engines.
Excellent work.
Vat
After reading this I found out that the shopping site I use (Shopify) does not allow me to modify/edit the robots.txt file. Would be helpful to know if there’s another solution to avoid having old files (from the prior shopping site host) indexed since they don’t exist any more.
I was just looking in to hiding a PDF, so thanks for the info.
One Question. Do All search engine spiders use the Robots.txt code in the same way?
I have a TON of pdf files that need hiding. Thank you so much!