Do 404s hurt my site?

So there you are, minding your own business, using Webmaster Tools to check out how awesome your site is... but, wait! The Crawl errors page is full of 404 (Not found) errors! Is disaster imminent??

Fear not, my young padawan. Let’s take a look at 404s and how they do (or do not) affect your site:

Q: Do the 404 errors reported in Webmaster Tools affect my site’s ranking?
A: 404s are a perfectly normal part of the web; the Internet is always changing, new content is born, old content dies, and when it dies it (ideally) returns a 404 HTTP response code. Search engines are aware of this; we have 404 errors on our own sites, as you can see above, and we find them all over the web. In fact, we actually prefer that, when you get rid of a page on your site, you make sure that it returns a proper 404 or 410 response code (rather than a “soft 404”). Keep in mind that in order for our crawler to see the HTTP response code of a URL, it has to be able to crawl that URL—if the URL is blocked by your robots.txt file we won’t be able to crawl it and see its response code. The fact that some URLs on your site no longer exist / return 404s does not affect how your site’s other URLs (the ones that return 200 (Successful)) perform in our search results.

Q: So 404s don’t hurt my website at all?
A: If some URLs on your site 404, this fact alone does not hurt you or count against you in Google’s search results. However, there may be other reasons that you’d want to address certain types of 404s. For example, if some of the pages that 404 are pages you actually care about, you should look into why we’re seeing 404s when we crawl them! If you see a misspelling of a legitimate URL (www.example.com/awsome instead of www.example.com/awesome), it’s likely that someone intended to link to you and simply made a typo. Instead of returning a 404, you could 301 redirect the misspelled URL to the correct URL and capture the intended traffic from that link. You can also make sure that, when users do land on a 404 page on your site, you help them find what they were looking for rather than just saying “404 Not found."

Q: Tell me more about “soft 404s.”
A: A soft 404 is when a web server returns a response code other than 404 (or 410) for a URL that doesn’t exist. A common example is when a site owner wants to return a pretty 404 page with helpful information for his users, and thinks that in order to serve content to users he has to return a 200 response code. Not so! You can return a 404 response code while serving whatever content you want. Another example is when a site redirects any unknown URLs to their homepage instead of returning 404s. Both of these cases can have negative effects on our understanding and indexing of your site, so we recommend making sure your server returns the proper response codes for nonexistent content. Keep in mind that just because a page says “404 Not Found,” doesn’t mean it’s actually returning a 404 HTTP response code—use the Fetch as Googlebot feature in Webmaster Tools to double-check. If you don’t know how to configure your server to return the right response codes, check out your web host’s help documentation.

Q: How do I know whether a URL should 404, or 301, or 410?
A: When you remove a page from your site, think about whether that content is moving somewhere else, or whether you no longer plan to have that type of content on your site. If you’re moving that content to a new URL, you should 301 redirect the old URL to the new URL—that way when users come to the old URL looking for that content, they’ll be automatically redirected to something relevant to what they were looking for. If you’re getting rid of that content entirely and don’t have anything on your site that would fill the same user need, then the old URL should return a 404 or 410. Currently Google treats 410s (Gone) the same as 404s (Not found), so it’s immaterial to us whether you return one or the other.

Q: Most of my 404s are for bizarro URLs that never existed on my site. What’s up with that? Where did they come from?
A: If Google finds a link somewhere on the web that points to a URL on your domain, it may try to crawl that link, whether any content actually exists there or not; and when it does, your server should return a 404 if there’s nothing there to find. These links could be caused by someone making a typo when linking to you, some type of misconfiguration (if the links are automatically generated, e.g. by a CMS), or by Google’s increased efforts to recognize and crawl links embedded in JavaScript or other embedded content; or they may be part of a quick check from our side to see how your server handles unknown URLs, to name just a few. If you see 404s reported in Webmaster Tools for URLs that don’t exist on your site, you can safely ignore them. We don’t know which URLs are important to you vs. which are supposed to 404, so we show you all the 404s we found on your site and let you decide which, if any, require your attention.

Q: Someone has scraped my site and caused a bunch of 404s in the process. They’re all “real” URLs with other code tacked on, like http://www.example.com/images/kittens.jpg" width="100" height="300" alt="kittens"/> Will this hurt my site?
A: Generally you don’t need to worry about “broken links” like this hurting your site. We understand that site owners have little to no control over people who scrape their site, or who link to them in strange ways. If you’re a whiz with the regex, you could consider redirecting these URLs as described here, but generally it’s not worth worrying about. Remember that you can also file a takedown request when you believe someone is stealing original content from your website.

Q: Last week I fixed all the 404s that Webmaster Tools reported, but they’re still listed in my account. Does this mean I didn’t fix them correctly? How long will it take for them to disappear?
A: Take a look at the ‘Detected’ column on the Crawl errors page—this is the most recent date on which we detected each error. If the date(s) in that column are from before the time you fixed the errors, that means we haven’t encountered these errors since that date. If the dates are more recent, it means we’re continuing to see these 404s when we crawl.

After implementing a fix, you can check whether our crawler is seeing the new response code by using Fetch as Googlebot. Test a few URLs and, if they look good, these errors should soon start to disappear from your list of Crawl errors.

Q: Can I use Google’s URL removal tool to make 404 errors disappear from my account faster?
A: No; the URL removal tool removes URLs from Google’s search results, not from your Webmaster Tools account. It’s designed for urgent removal requests only, and using it isn’t necessary when a URL already returns a 404, as such a URL will drop out of our search results naturally over time. See the bottom half of this blog post for more details on what the URL removal tool can and can’t do for you.

Still want to know more about 404s? Check out 404 week from our blog, or drop by our Webmaster Help Forum.

Source : http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html

Read more »

Page layout algorithm improvement

In our ongoing effort to help you find more high-quality websites in search results, today we’re launching an algorithmic change that looks at the layout of a webpage and the amount of content you see on the page once you click on a result.

As we’ve mentioned previously, we’ve heard complaints from users that if they click on a result and it’s difficult to find the actual content, they aren’t happy with the experience. Rather than scrolling down the page past a slew of ads, users want to see content right away. So sites that don’t have much content “above-the-fold” can be affected by this change. If you click on a website and the part of the website you see first either doesn’t have a lot of visible content above-the-fold or dedicates a large fraction of the site’s initial screen real estate to ads, that’s not a very good user experience. Such sites may not rank as highly going forward.

We understand that placing ads above-the-fold is quite common for many websites; these ads often perform well and help publishers monetize online content. This algorithmic change does not affect sites who place ads above-the-fold to a normal degree, but affects sites that go much further to load the top of the page with ads to an excessive degree or that make it hard to find the actual original content on the page. This new algorithmic improvement tends to impact sites where there is only a small amount of visible content above-the-fold or relevant content is persistently pushed down by large blocks of ads.

This algorithmic change noticeably affects less than 1% of searches globally. That means that in less than one in 100 searches, a typical user might notice a reordering of results on the search page. If you believe that your website has been affected by the page layout algorithm change, consider how your web pages use the area above-the-fold and whether the content on the page is obscured or otherwise hard for users to discern quickly. You can use our Browser Size tool, among many others, to see how your website would look under different screen resolutions.

If you decide to update your page layout, the page layout algorithm will automatically reflect the changes as we re-crawl and process enough pages from your site to assess the changes. How long that takes will depend on several factors, including the number of pages on your site and how efficiently Googlebot can crawl the content. On a typical website, it can take several weeks for Googlebot to crawl and process enough pages to reflect layout changes on the site.

Overall, our advice for publishers continues to be to focus on delivering the best possible user experience on your websites and not to focus on specific algorithm tweaks. This change is just one of the over 500 improvements we expect to roll out to search this year. As always, please post your feedback and questions in our Webmaster Help forum.

Read more »

Search, plus Your World

Google Search has always been about finding the best results for you. Sometimes that means results from the public web, but sometimes it means your personal content or things shared with you by people you care about. These wonderful people and this rich personal content is currently missing from your search experience. Search is still limited to a universe of webpages created publicly, mostly by people you’ve never met. Today, we’re changing that by bringing your world, rich with people and information, into search.

Search is pretty amazing at finding that one needle in a haystack of billions of webpages, images, videos, news and much more. But clearly, that isn’t enough. You should also be able to find your own stuff on the web, the people you know and things they’ve shared with you, as well as the people you don’t know but might want to... all from one search box.

We’re transforming Google into a search engine that understands not only content, but also people and relationships. We began this transformation with Social Search, and today we’re taking another big step in this direction by introducing three new features:

1. Personal Results, which enable you to find information just for you, such as Google+ photos and posts—both your own and those shared specifically with you, that only you will be able to see on your results page;
2. Profiles in Search, both in autocomplete and results, which enable you to immediately find people you’re close to or might be interested in following; and,
3. People and Pages, which help you find people profiles and Google+ pages related to a specific topic or area of interest, and enable you to follow them with just a few clicks. Because behind most every query is a community.

Together, these features combine to create Search plus Your World. Search is simply better with your world in it, and we’re just getting started.

Personal Results
Say you’re looking for a vacation destination. You can of course search the web, but what if you want to learn from the experiences your friends have had on their vacations? Just as in real life, your friends’ experiences are often so much more meaningful to you than impersonal content on the web. With your world in search, you can find:

• Google+ posts. You can find relevant Google+ posts from friends talking about an amazing trip they just took, whether they’ve shared privately with you or publicly. You’ll find links shared by your friends, such as activities, restaurants and other things they enjoyed on their trip.
• Photos. You can find beautiful vacation photos from your friends right in your search results page. You can also find your own private photos from Google+ and Picasa, based on captions, comments and album title.

Personal Results: a family story
As a child, my favorite fruit was Chikoo, which is exceptionally sweet and tasty. A few years back when getting a family dog, we decided to name our sweet little puppy after my favorite fruit. Over the years we have privately shared many pictures of Chikoo (our dog) with our family. To me, the query [chikoo] means two very sweet and different things, and today’s improvements give me the magical experience of finding both the Chikoos I love, right in the results page.

This is search that truly knows me, and gives me a result page that only I can see. And while I get a nice mix of personal results with results from the web, I can also click the link at the top of the results page (red arrow) for the option to search only within my world.

Profiles in Search
Every day, there are hundreds of millions of searches for people. Sometimes, it’s hard to find the person you’re looking for. Once you do find him or her, there’s no quick way for you to actually interact. Starting today, you’ll have meaningful ways to connect with people instantly, right from the search results.

Now, typing just the first few letters of your friend’s name brings up a personalized profile prediction in autocomplete. Selecting a predicted profile takes you to a results page for your friend, which includes information from their Google+ profile and relevant web results that may be related to them. And you can have this personal experience instantaneously, thanks to Google Instant. So when I search for [ben smith], I now find my dear friend Ben every time, instead of the hundreds of other Ben Smiths out there (no offense to all of them!).

In addition, you’ll find profile autocomplete predictions for various prominent people from Google+, such as high-quality authors from our authorship pilot program.

Once you select that profile, if you’re a signed-in Google+ user, you’ll also see a button to add them to your circles right on your search results page.

People and Pages
As I mentioned earlier, behind most queries are communities. Starting today, if you search for a topic like [music] or [baseball], you might see prominent people who frequently discuss this topic on Google+ appearing on the right-hand side of the results page. You can connect with them on Google+, strike up meaningful conversations and discover entire communities in a way that simply wasn’t possible before.

Unprecedented security, transparency and control
When it comes to security and privacy, we set a high bar for Search plus Your World. Since some of the information you’ll now find in search results, including Google+ posts and private photos, is already secured by SSL encryption on Google+, we have decided that the results page should also have the same level of security and privacy protection. That's part of why we were the first major search engine to turn on search via SSL by default for signed-in users last year. This means when you’re signed in to Google, your search results—including your private content—are protected by the same high standards of encryption as your messages in Gmail.

We also want to be transparent about how our features work and give you control over how to use them. With today’s changes, we provide interface elements and control settings like those you’ll find in Google+. For example, personal results are clearly marked as Public, Limited or Only you. Additionally, people in your results are clearly marked with the Google+ circle they are in, or as suggested connections.

We’re also introducing a prominent new toggle on the upper right of the results page where you can see what your search results look like without personal content. With a single click, you can see an unpersonalized view of search results.

That means no results from your friends, no private information and no personalization of results based on your Web History. This toggle button works for an individual search session, but you can also make this the default in your Search Settings. We provide separate control in Search Settings over other contextual signals we use, including location and language.

That's unprecedented transparency and control over personal search results.

A beautiful journey begins
Search plus Your World will become available over the next few days to people who are signed in and searching on https://www.google.com in English.

While there may be 7 billion people and 197 million square miles on Earth, a septillion stars and a trillion webpages, we spend our short, precious lives living in a particular town, with particular friends and family, orbiting a single star and relying on a tiny slice of the world’s information. Our dream is to have technology enable everyone to experience the richness of all their information and people around them.

We named our company after the mathematical number googol as an aspiration toward indexing the countless answers on webpages, but that’s only part of the picture. The other part is people, and that’s what Search plus Your World is all about.

Posted by Amit Singhal, Google Fellow

Read more »