The ability to collect and analyze organic search keywords has always been one of the fundamental capabilities of web analytics tools, in addition to one of the earliest ways to justify the effort of implementing web analytics. The search query was part of the referrer data collected on the initial request, and because it “just happened,” it was almost always taken for granted. At least it was until September 25, when Google – which accounts for 90% of search engine queries worldwide – announced it had begun blocking ALL organic search keywords to protect user privacy and not just those of users logged into their Google account.
What to do in a world where 90% or more of your organic keywords could be “not provided”? Well, the good news is that some keyword data is still available via other means, although not necessarily in a readily usable form.
Other Search Engines
So far, no other search engines have joined in Google’s crusade to improve user privacy by blocking organic keywords. That means you will still have some organic keyword data available going forward, just not the lion’s share of it. Websites in countries where Google is less dominant will fare better than those in the U.S.
However, it is likely that other search engines will follow suit, eventually. So make the most of your organic keyword data while you can.
Google Maps and Local Search
Google had already begun removing organic search query data from Google places search listings earlier in 2013, but clicks to locations within Google Maps are currently still passing the original query data (q=) as well as secondary query data (hq=, dq=) in some cases. This is part of the reason that you will continue to see some organic search keyword data originating from the Google engine.
TIP – If you have a lot of retail locations that are found via Google maps, you may want to modify your analytics solution to capture the value of “dq=” into a custom variable so you can track these keywords in a separate report. The example below shows where these links appear in Google Maps and the final query string that gets passed to the referral URL on the landing page.
Don’t get your hopes up too much though. It’s likely this loophole will eventually be closed, if that can be done without causing too much drama for the many users of the popular Google Maps API. Google only has to add a redirect, as they have done with everything else. Furthermore, users who actually click through from a map link to its website are typically a small fraction of a website’s organic search visitors.
In the meantime, this does mean that companies with many retail locations may get a bit more organic search keyword data than those who do not.
Google Webmaster Tools
Most sites will have a Google Webmaster Tools (GWT) account and organic search query data is still available within GWT, at least for the time being. The two biggest problems with this data is that Google Webmaster Tools only keeps data for the last 90 days and it only retains the top 2,000 or so terms. So, while you can compare GWT’s data with your most recent analytics data to gauge the mathematical relationship between the two tools, you only have a short window of time in which to establish that index. But prepare yourself to become more comfortable with uncertainty, as these will not be very exact numbers.
A third problem is that you will have to use click-throughs as the basis for that comparison and Google Webmaster Tools rounds those numbers into “buckets”, adding even more uncertainty to the mix. However, GWT does give you at least one way to use search query data to extrapolate organic keyword traffic for your site going forward, even if only directionally. Just make sure you go into GWT every 30 days and download all your search query data into CSV form. Those who saw the writing on the wall in 2011 when Google first began blocking keyword data and have already started doing this will be ahead of everyone else in making the best use of this data.
TIP – If you are using Google Analytics, make sure your Google Analytics profile is linked to your Google Webmaster Tools account. While Google has not announced any plans to remove historic organic search keyword data from Google Analytics, you may want to err on the side of caution and export at least a year’s worth of your historical data now, so you’ll have it for benchmarking later, regardless of what Google ends up doing.
Since this is Google we’re talking about, it is certainly possible that they will eventually block keyword data from Google Webmaster Tools at some point in the future. Many users reported that GWT was also not reporting search engine queries as of September 25, but Google has since said this was a bug. Data does seem to be populating again for many GWT accounts as of this writing, so presumably it was a bug.
Google AdWords and Paid Search
Fortunately, blocking keyword data does not apply to Paid Search and those companies who already rely heavily on Paid Search data to drive insight will have an advantage, or at least less of a disadvantage. On the other hand, companies who were at the top of organic listings and didn’t need to invest as heavily in paid search may feel like they are flying blind, at least initially. Even if you don’t plan to invest in paid search, you may want to set up a free AdWords account just so you can use Google’s Keyword Planning tool to assess the volume of queries for certain keyword phrases and identify opportunities for content optimization.
None of these tools are ideal, or will even come close to filling the gap left by Google’s removal of organic keyword data. The next question, then, is what can companies do to keep their initiatives that relied on organic keyword data moving forward, in spite of these obstacles?
Moving Past the Stumbling Blocks
Depending on how you’ve been using your organic keyword data, the impacts of this change could be profound, even daunting. Certainly it’s raised a ruckus in web analytics, content strategy, and SEO circles. However, for some use cases of keyword data, it seems like there may be ways to move forward without it.
Search Engine Optimization
For practitioners of SEO, some things will be business as usual, although it may not seem that way. Google AdWords’ Keyword Planner will still provide estimates of keyword opportunities that can be exploited for SEO, just as they are for SEM. Google Webmaster Tools will still provide keyword data on impressions and click-throughs that can give a reasonable approximation of traffic generated by specific keywords. The process of optimizing a site for crawling hasn’t really changed as a result of keyword blocking; it will just be harder to see keyword specific impacts as changes are made, especially when comparing brand vs. non-brand results.
The volume and variety of long-tail search phrases already made it much more difficult to gain insight from analyzing specific keywords. Broad match paid search has encouraged search engine marketers to focus on broad groups of high-value keyword categories, rather than on specific, lower volume keywords. This focus has bled over into SEO. Beyond the major brand terms, everything else is categorized at a threshold that makes it easier to execute an effective strategy around core brand messages.
As a result, there has long been a gradual shift in the industry to focus more on overall traffic acquired by organic search to specific landing pages and how those landing pages drive conversion – by relaying key brand messages and calls to action – rather than on the exact keyword phrases visitors typed in to Google. The same landing page data is still available from your analytics tool. You can still tell how much of your traffic was driven by organic search as a whole, you still know what pages they entered on, and you can still track conversion based on those entry points. You just won’t know the specific terms used.
For content managers and marketers, one of the most effective tactics will be to monitor trends in organic search traffic at the page level, rather than at the keyword level. Hopefully, you’ve already been doing that as you create content and use it to target certain visitors. Trending organic search arrivals at those entry points, and comparing that to search engine ranking data (via Google Webmaster Tools or other tools) will yield the best long term results in creating content that caters to the topics your visitors are searching for.
Another important tactic – and another which most sites were already using – is to mine the data generated by your onsite search tool. Create reports that will reveal not only the terms that your visitors are searching for, but also those they are searching for and not finding. These are areas ripe for some focused content and, as long as you follow the ground rules for optimizing your content for Google, visitors will find it.
One area of particular concern is the area of personalization, since it has become common practice to personalize the web experience based on the organic search terms used – basically a short cut to identifying the visitor’s needs. But it will no longer be possible to personalize based on Google’s organic search keywords. Doing so for the other engines – when they drive such a small volume of organic search traffic – is of questionable value. The deciding factor will be ROI. How much return must be generated from such an effort when it may only generate 1/10th of the volume it did in the past? And what happens if the other search engines decide to follow Google’s example and block their keyword data too?
A large part of the impact will fall on personalization efforts that relied on real-time or near-real-time keyword data to provide targeted, personalized experiences for new visitors. For those companies that are building personalization around user logins, customer IDs, or internal search data, there should be little disruption. There are also a host of other tools that can be – and have been – used to provide targeted experiences without relying on keyword data: Demandbase, Facebook, comScore, and Experian, just to name a few. Facebook Connect is of particular interest, due to the vast amount of user data they have at their disposal that can be shared at the user’s discretion and accessed by your site via API.
A Challenge, but also an Opportunity
While a world without readily available organic search keyword data may seem scary, in reality it may end up being a good thing for digital marketers and site owners, as well as for customers. Ultimately it will push us all toward optimizing based on engaging content and user behaviors that are further down the funnel, rather than on specific phrases that only indicate how a user found us, not what they were actually looking for. Those companies who can adapt to this new paradigm will also be those who can adapt to whatever new changes come along that we can’t yet anticipate.