According to robotstxt.org:

“In a nutshell: Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol”

caveat: “…don’t try to use /robots.txt to hide information.”