"Faster Hosting with FREE SSDs, Pre-Installed WordPress + Free Web Builder, 24/7 US Based Support"
Unlimited Bandwidth, Storage and Email, $150 Ad Credits with Bing, Yahoo, and Amazon, 20x Faster Hosting
Find out what is robots.txt and how it can benefit your site’s SEO. Power your website with Hostinger web hosting! 👉
💥 Use the discount code HA10 to get 10% OFF!
Optimizing a website involves more than keyword research and on-page optimization. Having a robots.txt file will make your site’s crawling and indexing process more efficient, presenting searchers with more relevant and useful content. If you’re new to website optimization, watch this video to learn what is robots.txt, how it works, and how to create one for your website.
____________________________________________
📚 Get Our eBooks!
WordPress Cheat Sheet:
7-Day WordPress Course:
____________________________________________
📌 Handy Links
The Complete Guide to WordPress robots.txt – Hostinger Tutorials:
How to Create a Sitemap in WordPress –
How to Connect to FTP Server Using FileZilla –
Join Hostinger Affiliate Program:
____________________________________________
⭐Follow Us⭐
Instagram:
Twitter:
Facebook:
WordPress Explained Group:
Hostinger Tutorials:
Subscribe to our channel:
____________________________________________
🕒 Timestamps
____________________________________________
00:00 – Introduction
00:30 – What Is a robots.txt File and Why Do You Need It?
02:13 – How Does robots.txt Work?
03:52 – Creating a robots.txt File in WordPress
04:10 – Using Yoast SEO
05:34 – Using AIOSEO
06:37 – Using Hostinger’s hPanel
08:07 – Using an FTP client
08:41 – Testing the robots.txt File
08:56 – Outro
___________________________________________
🟣 What Is robots.txt and Why Do You Need It?
robots.txt is a file consisting of instructions for search engine bots. It tells them which directories they should or shouldn’t crawl.
🟣 How Does robots.txt Work?
WordPress will automatically generate a virtual robots.txt file with a default syntax. You will need to create your own to add more rules or edit the existing ones. Learn how you can do that in this video.
🟣 Creating a robots.txt File in WordPress
We will cover four different methods of adding a robots.txt file to your WordPress website:
👉 Using Yoast SEO
👉 Using AIOSEO
👉 Using Hostinger’s hPanel
👉 Using an FTP client
🟣 Testing the robots.txt File
Every time you modify the robots.txt file, test it first to avoid mistakes that can affect your website’s SEO. Watch how to do that using Google Search Console.
If you have any questions about what is robots.txt or how to create it, leave us a comment.
___________________________________________
▶ Want to see more awesome tutorials like this in the future? Consider subscribing 😁
https://www.YouTube.com/c/HostingerAcademy/?sub_confirmation=1
▶ If you have other questions, don’t hesitate and join our Facebook group for more help:
____________________________________________
Thank you for watching! Let us know in the comments below if you have any questions. Good luck on your online journey. 🚀
#whatisrobots.txt #SEO #HostingerAcademy
Hi I'm Victoria and in this video we are Going to discuss what is robots.txt file How it works and how to create it on WordPress website let's go To better optimize your website you need To ensure that search engine Bots can Crawl your most important pages to help You in that process there is a file Called robots.txt it helps direct search Engine Bots to the web pages you want Them to index Roberts.txt is a file containing Instructions for search engine robots Telling them to crawl or avoid web pages Uploaded files or URL parameters in Simple words the robots.txt file tells About crawlers hey you can look at this Part of the website but don't go there To understand how it can benefit your Website's optimization let's talk about The search engine crawling process when Someone creates a new website search Engines send their crawlers to discover And collect information required to Index a page once web crawler find Information such as keywords and fresh Content they will add the webpage to the Search index when users perform a search The search engine will fetch the Relevant information from indexed Websites without the robots.txt file the Bots May index Pages not meant for Public view or even fail to visit your Most important Pages plus modern
Websites contain any more elements than Just web pages for instance if you're Using WordPress there's a high chance That you will install plugins doing that Also means downloading the plugins Directory which might show up in your Search engine results even though they Aren't relevant to your website content More importantly without a robots.txt File there can be too many Bots crawling Your website and slowing down its Performance that's why I highly Recommend having a robots.txt file on Your website creating it won't take long And you can make the file as detailed as You want unfortunately maintaining it is Also easy even if you don't have any Technical knowledge When you create a WordPress website it Will automatically generate a virtual Robots.txt file located in your site's Main folder you can see your website's Robot.txtip file by adding slash Robots.txt after its domain name if you Are curious about a certain website's Robot.txt file you can view it using the Same method you'll see a file similar to This that's a basic robots.txt syntax Which in this example consists of three Lines of rules you can add or remove any Rule according to your site's needs the Part after user agent declares for which Bots the rules below apply to in this Example we can see an asterisk which
Means the rules apply to all crawler Bots if you want to create a rule for a Specific search but replace the Asterix With the name of the bot for example Google bot the rules below are both Disallowed directives each followed by a Website directory in in short this Roberts.txt file tells all Bots that They shouldn't crawl your WP admin and WP includes directories those folders Contain many sensitive files that your Audience shouldn't see so it makes sense To have those rules besides that you can Add as many rules as you want for Example Amazon's robots that txt file This allows all Bots to crawl several Pages however the robots.txt file that WordPress setups for you by default Isn't accessible if you want to edit it You will need to create your own file And upload it to your root folder Now that you know how a robots.txt file Works I'll show you how to create your Own file with four different methods Using yoast SEO and all in one SEO Plugins or manually via hpanel and FTP Client If you have already installed yoast SEO You can go directly to the tools menu But if you haven't install and activate The plugin first hover over the plugins Menu and choose add new type in yoast SEO on the search bar click install and Once it's complete activate now you can
Configure your site's SEO setting after Completing the installation wizard go to The tools menu find the option that says File editor it lets you edit HD access File directly on the WordPress dashboard Now let's click on create a robots.txt File it will open a text editor where You can tweak it note that the directive Lines like user agent allow and disallow Are not case sensitive but the directory Values are case sensitive so pay close Attention when writing its Syntax for Instance product in lowercase indicates A different path and product in Uppercase also typing a single slash as A directory value means your entire site Meanwhile leaving it blank means nothing Let's see an example if you write this Rule it means you are blocking all Search engine Bots from crawling your Entire site meanwhile if you leave it Blank you're allowing all Bots to access All sections of your site now let's move On on to another method If you prefer using all-in-one SEO Plugin the process of creating a Robots.txt file is as easy to access All-in-one SEO dashboard you will need To install it activate the plugin and Complete the configuration wizard it Will automatically import the default Robots.txtip file generated by WordPress To add specific rules for your site go To the tools menu
Switch on the option enable custom Robots.txt to open the file editor All-in-one SEO has a beginner friendly Interface making it easy to create rules To add a rule start by entering the user Agent next choose either the allow or Disallow directive then add desired Directory path if you want to add more Rules just click the add rule button and Repeat the previous steps once you're Done click save changes the rules will Be automatically added to the robots.txt File which you can view by clicking the Open roberts.txt Button Another option is creating a robots.txt File manually and uploading it via Hostinger's hpanel or an FTP Client First you need to create the robots.txt Using text editor software like notepad Or text edit using the syntax I showed You earlier in the video there's no Limit to how many rules you can add so I Recommend adding the sitemap director Which will tell search engines the Location of your XML sitemap file an XML Sitemap lists a website's essential Pages so that search engines can find And crawl them and helps them understand Your website structure as you want to Make sure search engine scroll your most Important web pages if you use an SEO Plugin like yoast or all-in-one SEO it Will automatically create the sitemap Directive once you set up an XML file if
You want to learn more about what is a Site map and how to create one make sure To check video I Linked In the info card After listing all the rows save it as Txt file note that the file name is k Sensitive so you must name it robots.txt All in lowercase now that you have the File access your hpanel dashboard and go To file manager open the public HTML Directory now drag the robots.txt file From your local computer and drop into The public HTML folder the uploading Process will only take a few seconds You can also use an FTP client to upload Robots.txt file start by connecting your Website to an FTP client like pilezilla If you need guidance on how to do that I Included a video tutorial in the Description below once you're connected To your site via FTP navigate to the Public HTML folder all you have to do is Upload the robots.txt file from your Local computer to your web server by Dragging and dropping it into the folder It should take only a few seconds for The uploading process to complete Make sure to test your robots.txt file Every time you change its content to Prevent any mistakes from affecting your Site's SEO Google search console is Great for that as it includes a Robots.txt checker that's it you'll Learn what the robots.txt file is how it Can benefit your site's SEO and how to
Create it if you have any questions Don't hesitate to leave them in the Comment section below leave a like if You enjoy this video and for more videos About SEO and website development Subscribe to hostinger Academy good luck On your online Journey Foreign
