How Important is robots.txt file

#robots #robotstxt