What Is The Difference Between Crawling and Indexing?
There are many terms that are thrown around all the time inthe SEO world, many that seem to be synonymous. One perfect example of terms that are incorrectly used synonymously is Crawling and Indexing. Whether the writer understands the difference or not, in many SEO articles it leads the reader to believe the two words mean the same thing. They most definitely do not. So…exactly what is the difference between crawling and indexing?
What is Crawling?
Crawling or spidering is a term used when Google, or another search engine, sends its bot to a web page or web post and “reads” the page. Don’t let this be confused with having that page being indexed. Crawling is the first part of having a search engine recognize your page and show it in search results. Having your page crawled does not necessary mean your page was indexed and will be found. Pages get crawled for a variety of reasons…the most common is having an XML sitemap that Google reads, which points to your new page.
You should have an XML sitemap uploaded to Google Search Console (formally Google Webmaster Tools), giving Google the road-map for all of your new content. Other ways Google will crawl your page is your page may have been posted to Google+ and Plus 1d (which sends the Google bot), or Google may have simply run across your page and crawled it. What getting crawled means is Google has looked at the page and depending on if Google thinks the content is “New” or otherwise has something to “give to the Internet” it may schedule to be indexed. Also, when Google crawls a page, it looks at the links on that page and schedules the Google Bot to check out those pages too. In no way does having your page crawled mean that it has been indexed and would even has a chance to be found in a Google search.
What Does Being Indexed Mean?
Having your page Indexed by Google is the next step after it gets crawled. By no means does every site that gets crawled get indexed, but every site indexed had to be crawled. If Google deems your new page worthy, it will index it. Upon your page getting indexed, Google then comes up with how your page should be found in their search. Google then decides what keywords and what ranking in each keyword search your page will land. This is done by a variety of factors that ultimately make up the entire business of SEO. Also, any links on the indexed page is now scheduled for crawling by the Google Bot. It’s not only those links that get crawled, it is said that the Google bot will search up to five sites back. That means if a page is linked to a page, which linked to a page, which linked to a page which linked to your page (which just got indexed) then, they all will get crawled. This is the basis of why external links that come to your site are so important. The higher quality of the page that ultimately links to you, the better you will rank in the all powerful Google Search. This is what many SEO companies charge big money for—creating (or allowing the creation of) many links coming to your site from hi-quality websites using keywords you want to be found by. Its not the ONLY thing that an SEO Company may do, but it’s almost guaranteed to be on the list.
How Can I Tell What Google Has Indexed?
Although you NEED your site to be crawled, you WANT it to get indexed. There are a couple of ways to determine what Google has indexed on your site. One is to simply go to Google.com and click on Settings at the bottom right then choose Advanced Search. From there, scroll down to “site or domain” put in your website and hit Search. This will show you everything that Google has indexed. It should include pages, posts as well as photos and possibly other things such as feeds. The preferred way to see exactly what Google has indexed, because you have some control over fixing it, is to use Google Search Console (previously named Google Webmaster Tools). We are not covering how to set up Google Search Console in this article, but if you have a website, it NEEDS to be done. Google Search Console lets you upload an XML Sitemap which lets you tell Google what you would LIKE them to index and how often they should check back for changes. Google Search Console also provides a ton of valuable information on your website and is really the only two way communication with Google that exists.
How Does Google Decide What To Index?
This is the real question everyone should be asking. At the end of the day, Google will index new, fresh content that Google believes will improve the user experience of THEIR clients—that is the people that go to Google and search for something. They are very picky about trying to provide the most relevant websites for a specific search term. If you are copying pages, or using copy that is otherwise already in their index, there is no need to index yours. You may have heard the term “Duplicate Content” thrown around in SEO articles. Duplicate content is a point of contention for many SEO gurus, I personally say: at best it confuses Google on which page to rank, at worst you get penalized. At the end of the day, stay away from duplicate content. But I digress… If what you have written is BETTER or provides more information or Google otherwise believes that showing your page as opposed to the other pages will give their clients a better experience, they will index and rank your site. This is why providing fresh, new SEO rich blog content is so important. The more pages indexed with internal links to other pages within your site the better for SEO.
YaY! Now I Understand SEO!
We are just scratching the surface of what Google likes or how to effectively leverage SEO. Depending on your type of business, there are different ways to have your company found in a Google search. For instance, if you are a bricks and mortar type of business with a storefront, you will want to focus on Local SEO. Local SEO focuses on searches that include a city or location. For instance, if you wanted to find an SEO Service in New Orleans, you would Google New Orleans SEO That type of search will provide you with local results for a Search Engine Optimization Company.. If you are a dry cleaner, needless to say this type of searching is important to you. If you provide online training, your geographical location is not so important.
If you dig on this article, you can sign up to get other cool stuff like this directly in your inbox! Just look at the very bottom of the page to the left for the red box. If you are familiar with RSS feeds, you can get our last ten articles HERE.
Help Spread The Word...