Adding a dynamic robots.txt to Kohana 3

Adding a function that dynamically generates a robots.txt to Kohana 3 is actually very easy.

First we need to add a route to bootstrap.php that will route the yourdomain.com/robots.txt url to the function.


Route::set('robots', 'robots.txt')
->defaults(array(
'controller' => 'home',
'action' => 'robots',
));


Now we create the action "action_robots" in the home controller and output the exclusions we are looking for.


public function action_robots()
{

header('Content-type: text/plain');
echo "User-agent: *\r\n";
echo "disallow: /";
exit();
}


The content-type here is very important, as search engines will not read it as a php file.

NOTE: This will not work if your application if the index_file configuration in bootstrap.php is set to TRUE. This is because if it is set to TRUE then the url would turn out to be /index.php/robots.txt which is obviously not where the search engines will be looking for the restrictions.