If you want to start blogging professionally, you need to know some technical details related to SEO. New bloggers often start their blogging journey by submitting content to Blogger. It is important to enable a custom robots.txt file on these platforms. People often wonder what the best custom robots.txt file settings are for their blog. Let's learn everything today!
Robots.txt
Robots.txt is a simple text file that tells crawlers (also called spiders or bots) which parts of a website or blog they should crawl and which parts they should not.
How important is the Robots.txt file?
The robots.txt file is important for website owners and administrators. The success of any blog usually depends on how Google ranks your blog. It serves as a communication tool between the website and web crawlers (also known as bots or spiders) that index web pages and rank them for search engines. Often times we don't want Google to index all of these components. For example, you may have a file for internal use that is not useful to search engines. You do not want this file to appear in search results. Therefore, it is wise to hide these files from search engines. However, its importance can vary depending on the site's goals and structure.
Robots.txt file contains directives that all the top search engines honor. Using these directives you can give instructions to web spiders to ignore certain portions of your website/blog.
Robots.txt file is for Blogger/Blogspot blog
Since Blogger/Blogspot is a free blogging service, your blog's robots.txt file is not directly under your control. But now in Blogger, it is possible to make changes and create a custom Robots.txt file for each blog. The Robots.txt file for a Blogger/Blogspot blog usually looks like this:
User Agent: *
Disallow:
User Agent: *
Disallow:/search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
Add your custom Robots.txt file to Blogger/Blogspot
- Go to your blog's dashboard
- Open Settings > Search Preferences > Crawlers & Indexing > Custom robots.txt > Edit > Yes
- Here you can make changes to the robots.txt file
- After making changes, click the Save Changes button
To view your blog's current custom robots.txt file, go to the following URL:
http://www.yourblog.blogspot.com/robots.txt
The following wildcards are often used in robots.txt files.
* It means everything, everything
/ means the root directory
User agent
This directive indicates to which web crawlers the settings in the robots.txt file will be applied.
Disallow
It instructs web crawlers not to crawl the indicated directory or file. For example
Disallow: / will tell the web crawler not to crawl anything on your blog (because you don't allow the root directory).
Disallow: /dir/* will instruct the web crawler not to crawl any file under the /dir/ directory.
Disallow: /dir/myfile.htm will instruct the web crawler not to crawl myfile.htm under the dir folder. The crawler will crawl all other files under the dir directory.
Allow
This directive specifically tells the web crawler to crawl a specific directory or file. For example:
Disallow: /dir/myfile.htm
allow: /dir/myfile.htm
The general meaning of the above two lines is that the crawler should crawl /dir/myfile.htm
Site Map
A sitemap is a file or page on a website that provides an organized list of URLs (uniform resource locators) that help search engines and website visitors navigate and understand the website's structure. There are two basic types of sitemaps: XML sitemaps and HTML sitemaps.
Sitemap is a very important file in your website/blog. This file contains the structure of your website. It helps web crawlers find their way through your blog. Sitemap: The directive tells the crawler the location of your sitemap file. In the case of Blogger/Blogspot — you can leave this line as is.
However, it's important to note that not all web crawlers strictly adhere to the rules specified in the robots.txt file, and malicious bots may ignore it altogether. Therefore, while robots.txt is a valuable tool for controlling how search engines interact with your site, it should not be relied upon as a sole means of protecting sensitive information or preventing unwanted access.
That's it! If you have any questions regarding Blogger/Blogspot custom robots.txt file, let me know in the comments section. I will do my best to help you. Thank you for your interest
Comments
Post a Comment