Googlebot control

Control search engine robots You will learn: how to control the search engines robots   CONTENT 1. Let the search engines robots robots.txt ...

Control search engine robots

You will learn: how to control the search engines robots

 

CONTENT

1. Let the search engines robots robots.txt

2. Robots Meta Tag use

3. checking indexation

4. Forced search engine robots Call

1.Let the search engine robots to your website

Despite the fact that modern search engines (such as Goglia, Yahoo, Bing, etc.) extremely well and very quickly indexed (indexed, this means that a special search engine reads your website content so that it presented to people searching for information online Query list; search google for example .: entering the desired search in Google will provide a list of sites (in addition to just crawl index your site appear in the search list) to all new and existing sites and online stores, it is still appropriate to further specify their website or online store, where and how the search engines robots are invited to visit you.

What do the search engines robots indexed / visited your site?
You need to create a special file (robots.txt) and uploaded to your server's root directory. If you do not know how to do this please contact the website or e-shop programmer. As before indexing "robots", in particular, check whether there is a robots.txt file, then this file according to the current guidelines site is indexed or indexed or indexation only by some kind specified directories or files.

How to check whether you have the search engine robots to file?
Enter your website address (domain), and the end add "/robots.txt. Example: https://seositeoptimization.com/robots.txt
You should see something like this:

• User-agent: *
  Disallow: / admin /
  Disallow:
  Sitemap: https://seositeoptimization.blogspot.com//index.php?route=information/sitemap

• User-agent: * - means that the site is introduced by all search engines.
• Disallow: / - the symbol "/" we indicate what we do not want to index.
• Disallow: - without the sign "/" what's indexed. And in this case it is best to specify.

Disallow: /admin/ - It means that the search engine robots to index is banned admin directory, which is designed to store electronic information management. This is the e-shop CMS. Why ban the index? For you there is no point, that the administrative management of robots indexed files. Plus the fact that to avoid duplication of information (who does not like google) and for other similar reasons. Although in fact Google is intelligent enough system and in practice it does not index admin directories, but in any case, it is advisable to prohibit directory indexing

[ads-post]

Two things, attention robots.txt:

• Search Engine robots can ignore your robots.txt files in the current guidelines. Especially those robots that scans the Internet for Internet security and vulnerability and such robots (spec. Program), which collects the purpose of spamming e-mail addresses will ignore robots.txt.
• The robots.txt file is accessible and visible, so everyone can see what you've set (eg .: not to index any file server).

Thus, the robots.txt file is not intended to fully paslėti you want from the search engines for some information.

Additional examples of how sukofiguruoti robots.txt file

Block all search engines robots to index the entire site / server:

• User-agent: *
  Disallow: / 

Allow all search engines robots to index:

• User-agent: *
  Disallow:  

(Default is allowed to be indexed, so in this case can be and not to create a robots.txt file or just leave it blank)

PARTLY PROHIBITING index the Director

• User-agent: *
  Disallow: /cgi-bin/
  Disallow: /tmp/
  Disallow: /junk  

prevent specific search engine robot

• User-agent: BadBot
  Disallow:  

ALLOW ONLY one indexed search engine robots

• User-agent: Google
  Disallow:
  
  User-agent: *
  Disallow: /

PROHIBITING index of all server directories (folders), except for one It's a bit tricky because there is no function "Allow". The simplest way is to all of the files that are not indexed placed in a separate directory such as "stuff", but other files leave one equals above for "suff" directory:

• User-agent: *
  Disallow: /~joe/stuff/

Prevent specific files:

• User-agent: *
  Disallow: /~joe/junk.html
  Disallow: /~joe/foo.html
  Disallow: /~joe/bar.html 

2.Robots Meta Tag use

The site code include a special meta tag that looks like this:

• <meta name="robots" content="INDEX, FOLLOW" />  
  
  

Where to include and how to do it?
This code must be inserted into the head section of the online shop, which is derived as follows: </ head>
This meta name tells you that the website is open for search engines to index (that the search engines robots are allowed to visit the specified website. If needed additionally you can specify how often the search engines crawlers to visit your site the next time by adding the following code:

• <meta name="revisit-after" content="7-days " />
  However, this encoding is useful only when you are on a weekly (or the frequency) update your site or add a lot of new information (which is more suitable for news, articles, catalogs). However, if you regularly accommodates new information (of course it has to be unique and does not copy) to the site will be indexed is also quite good. Because the search engine robots when they saw that the site is constantly revolv- (which search engines love) in the case of robots, and themselves often visit your site.
  

  3.checking indexation

  How to check whether your website or e-shop index?

Jeigu svetainė buvo suindeksuota tuomet google paieškų sistema pateiks visus Jūsų svetainės puslapius, kurie buvo suiIf a site has been indexed if the Google search engine can find your web pages that have been indexed. And if you have not been indexed then not make any of your site pages
And if you would like to check that your e-shop or web site page is in the best position for certain keywords then google search, you should write as (but best to do it in a box out of the Google use):

4. Forced Search Engine Call

Take advantage of Google Webmaster Tools opportunities:
Another way to bring up the search engine robots to visit your website or online store. Take advantage of Google Webmaster Tools. Register and verify your site. To access the Search Engine robots to go to: Health Fetch as Google
This gives Google the right to call up the search engine robots to a specific page on your site. Index will not only given the options, but also all related pages (that all your website pages