Have you ever wondered what is Googlebot, and what role it plays in the positioning of your website using the internal links in seo?
Would you like to know what are the aspects that you must take into account that will make knowing better the robot of Google?
Did you know that there are seo tools that allow you to simulate the bot to Google how, for example, Screaming Frog?
The Google algorithm and, in short, the robots that crawl the sites are based on a lot of factors that determine whether your website will position better or worse in the page of results, since the relevance of the content to its quality, passing by a myriad of technical issues (whether you have a website and information literacy as a blog, or you want to do seo in e-commerce).
In fact, within the SEO optimization, there are factors that are minimum requirements, and other that will be the that will have you to compete. All of them are needed to achieve top positions on Google, but we can not start the house by the roof.
In the following pyramid shows the most basic elements of SEO; on the base of the pyramid are the fundamental elements to get it to appear in Google and on them that will help your website to be competitive:
In today's article we will talk about the most basic part of SEO, those fundamental elements that, if not taken into account, may prevent your website from getting to position in Google: the crawling and indexing.
Difference between crawling and indexing
In the first place, it is necessary to understand these two concepts. Although the crawling and indexing tend to go hand in hand, are two different phases within the process that follows Google to include the content from your site in their index. What are they?
Crawling is the process that follow Google and other search engines to discover your website. To do this, use robots that navigate by the web site through the links, that in the case of Google, it receives the name of “Googlebot”.
That is to say, tracking is the method followed by the search engines to navigate your website. On the other hand, the indexing is the process by which search engines to include a website in SERPs of Google.
For example, Google can crawl a website and not to index it, that is to say you can navigate it but not save it.
How does Googlebot?
These are the steps that the bot Google continues to track our website:
- When Googlebot comes to your site, begins to follow all internal links to find your content.
- It analyzes the content of the pages that have been crawled.
- Makes a copy of your website then stores it in its index.
- Lists the contents depending on the subject matter.
- Gives a value to the web based on its content.
When the user makes a search on Google, through the algorithm Google gives a ranking of the results that will best fit your search:
Why a page of your website is not indexed?
There are several reasons that a URL of your website is not indexed in Google:
The URL is blocked in the file robots.txt.
The file robots.txt in a file that indicates to the search engines what URLs can or can't access. If a URL or set of URLs are blocked in this file, Google does not crawl.
Noindex meta tag
It is a tag in the html of each page of the site that indicates if a page should or should not be indexed and if your links have to be, or not followed by the search engines.
Shown in the following way:
That will be indexed and the search engines will follow the links to discover other pages.
That will be indexed and the search engines will not follow the links on that page.
Not indexed but the search engines will follow the links to discover other pages.
Nor indexed, or follow the links.
Bound
If a url is not linked from any site, it is difficult for Google to find and index.
Content in Javascript: If a url is in Javascript, Google may have problems in your crawl that also affect the indexing.
Google and JavaScript
JavaScript has become, without a doubt, the main language of the web, but Google has always had problems to trace it and run it properly. Although today, the Internet giant has evolved a lot in this sense, it still has some problems.
This is not to say that a web in JavaScript will not be able to position, but that Google will cost you something more to index it.
What can you do if your website is in Javascript?
Your web JavaScript can be processed on the server or directly in your browser. Depending on how you do it, it will be more or less difficult for Google to crawl it.
- Server Side Rendering: a site can be created in Javascript, but be configured to run on the server and, when the website is loaded in the browser (eg, Chrome) do it as a website in html, a language that is much easier to understand for Google. From the point of view of SEO, this is the recommended option as it makes the web work faster both for users and for search engines.
- Client Side Rendering: unlike in the previous case, JavaScript will be loaded directly in the browser, that is to say, for Google it would be more difficult to crawl the web.
How processed by Google JavaScript?
The process of indexing of JavaScript is done in 2 phases:
- Googlebot crawls the web: Googlebot accesses a URL, but first see the file robots.txt to make sure that you can keep track of. Then, through the links to see the URLs linked (except if it tells you that you should not follow them). If the page is Server Side Rendering (that is to say, it is processed on the server) there is no problem and the indexes.
- If the page is Client Side Rending, that is to say, if running in the browser, Let Google the URLs in the queue, and expected to have more resources to run them. Googlebot crawls the page has already been executed (in html) and finally the indexes.
As a conclusion, we must take into account that if Google can't track correctly, the web will be much more difficult index and, therefore, position. Remember, the internal links will be fundamental for the correct tracking of your website and keep in mind that if your website is in JavaScript, you'll need to speak with the technical team to make sure of their proper indexing.
