The robots.txt file is a simple text file located in your root Joomla directory. It's purpose is to tell search engines which directories it may and may not spider. We can also add an entry to tell search engines where your sitemap is explicitly located. This helps make sure that all your products are easily found and indexed by the search engines. RedMystic proactively notifies Google, Yahoo and Bing when your products change but adding the sitemap to your robots.txt file ensures that all other search engines easily find your sitemap.  Requires RedMystic Core Component 1.0.7+

On many linux environments RedMystic will not have necessary privileges to automatically update your robots.txt file. You can either manually add the sitemap to the robots.txt file or change the permission of the file.

Changing permissions and letting Redmystic manage it:

1. ssh or ftp/sftp to your server with root privileges.

2. Navigate to the your root Joomla directory.

3. chmod o=rwx robots.txt

Manually changing your robots.txt file:

1. ftp to your server

2. Navigate to your root Joomla directory

3. Download your robots.txt file to your local machine

4. Edit the local robots.txt file and add an entry to the bottom of the file for the sitemap.  The complete robots.txt file should look like this:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Sitemap: http://<Your Domain>/index.php?option=com_redmystic&view=sitemap&format=raw&layout=raw&sitemap=sitemap.xml