Check your robots.txt file
Go towww.<i>yoursiteaddress</i>.com/robots.txt. You might get a
Page Not Founderror. That’s okay for your purposes: A no
robots.txtfile means you’re not placing any broad limits on what search engines can and can’t index on your site. You might also see something like this:
User-agent: * Disallow: /blog.htmThis file is called the
robots.txtfile. It tells search engine crawlers, also known as robots, what to do when they visit your website. In this example, it’s telling all search engines to ignore the blog.htm page. All other pages are searchable.
If you want to become a
robots.txtgeek, visit the Web Robots Page. You can find out everything you ever wanted to know about guiding robots around your site.
What you don’t want to see in your
robots.txtfile is this:
Disallow: /This line tells a visiting search engine crawler to ignore every page on your website. A developer may add this line when he is building the site to prevent search engines from crawling it while it’s under construction. If it is left there by accident, your site is invisible to search engines.
If your
robots.txtfile has any
Disallowcommands in it, check with your webmaster or developer to make sure that a reason exists.
Disallowcan be used to hide pages that change a lot, hide duplicate content, or keep search engines out of stuff you just don’t want them crawling. Just make sure that you’re not accidentally hiding content they should see.
Check for meta robots tags
Using the metarobotstag is another way to hide pages from search engines. Go to any page on your website and view the source code. You don’t want to see
<meta name="robots" content="noindex,nofollow">If the meta
robotstag is there, and it contains
noindex,
nofollow, or both, remove it. You have valid reasons to use this tag: You might want a search engine to ignore this page because it’s a duplicate; you might feel the information on the page is inappropriate for search results; or the developer might have hidden the page during development. If you don’t know the reason, delete the tag.
Do not trust your developer to remove the meta
robotstag. When he builds your site, he’s working hard, writing code so fast that his fingers smoke. Forgetting to remove that one little line of code is easy when you’re facing a tough deadline and still have 4,000 lines of code to write. Remind your developer!