robots.txt is a text file in your site's root that tells search robots which sections they may crawl and which to leave alone. It is the first thing a robot reads when it visits your site.
robots.txt keeps robots from wasting effort on junk pages - the cart, account area, technical sections. It also points to your sitemap so search engines find all your pages faster. Without it a robot still crawls the site, just less efficiently.
robots.txt does not hide a page from people and does not guarantee it stays out of search results. If other pages link to a blocked page, Google may still show it. For real hiding you need other tools - a password or a noindex meta tag.
A free 30-second audit shows whether everything is fine on your site.
Check your site for free