Lessons Learned Using Google Consumer Surveys

There are a lot of things I enjoy, but one that tends to make me smile is when I get to play with other people’s money. This is especially true if the donor is Google. Their coupon offer to use Google Consumer Surveys was one I couldn’t resist, but I should’ve waited.

The service appealed to me for two reasons:

  1. As a marketer, I thought I could use a survey to get an idea of what topics prospective webmasters might like to learn.
  2. As a publisher, I was curious whether I could use surveys as a way to monetize various web properties.

The idea behind Google Surveys is simple. Instead of publishers putting up a pay wall, you show viewers about 100 words of an article with the option to take a survey to reveal the rest. In Google’s whitepaper, they refer to this as a “surveywall”.

If readers don’t wish to take the survey, they can complete an alternate action to see the content. The alternative action might be a social media share or email signup.

Although I wanted to try the service, I hadn’t seen any surveys in the wild through my normal web surfing. It took some digging to find sites where these surveys were in use. This could be that I didn’t meet the survey requirements so they did not display. Below is an example I found on HitchedMag.com.

Example of Google Consumer Survey
Survey from HitchedMag.com

The top section [1] is where you see the article lead. To read the remainder, you either Answer a question [2] or Share the page you’re reading [3]. I answered “yes” was then asked which of two brands I prefer along with product images. However, because I’ve never tried either brand, I opted to get a new question. Simple and fast.

What’s different about these Google surveys is they are only 1 or 2 questions by design. This limits survey fatigue and abandonment, but it also means you can’t do certain types of analysis like crosstab. You can create surveys with multiple questions, but the system allocates the questions so a recipient will only see one of those questions. According to Google, this increases the completion rate.

In my case, I wanted to find out what topics appealed to webmasters. I could put a WordPress poll on this site, but that would limit me to people who were already read the site. I wanted a broader audience. I was also hoping that like Google AdWords Placements, I could choose which web properties displayed my survey; however, there isn’t that mechanism.

Creating the Survey

For my survey, I used a screening question because I wanted to target webmasters. This is the more expensive of the 2 options, because you’re excluding people that don’t meet your criteria with the first question. Surveys with screening questions cost 50 cents a respondent versus 10 cents. You can also screen by age, gender and geography. One limitation is the surveys are only available in the US, Canada and the U.K.

Another caveat is the surveys don’t work with Apple’s Safari according to an interview with the developer, Paul McDonald. At first, that might make you pause but if you look at the current browser stats, the percentage is currently at 4.1%. Of course, if you’re selling to people who just use iPad or iPhones that’s a different story.

On my first attempt, my screening question was “Do you run or manage a website for a small business?” If someone answered “yes” to that question, they saw the second question. The problem I encountered is too few people answered “yes” so Google stopped the survey and refunded my money. I thought I would have an opportunity to edit the survey, but you have to buy and create a new survey. You do have the option of copying an existing survey.

I wasn’t sure how to correct the problem so I emailed support. I figured if I asked something like “Do you have a webmaster tools account with either Google or Bing?”, I would get fewer people. I also wasn’t sure if I could use brand names so I ended up removing the word “small” and the survey completed. You might want to look at some of the example surveys to see how people phrased their questions.

My second question dealt with specific topics that interested webmasters. I asked “Which subject do you wish you knew more about?” At the time I created the survey, I missed an important footnote about this question type. The system allows you to create many answers from which people select one, but the viewer will only see 5 choices.

For people taking my survey, there was always one answer they didn’t see. I’ve highlighted the picture below to show where this important note is so you don’t make the same mistake. If you’re running a large survey with thousands of respondents, this probably wouldn’t make a difference, but I was only asking for 200. I should also mention there are a dozen different question types.

footnote explaining 5 answer limitation
Ooops. I Wish I Saw This Note.

The survey system randomizes the answer order and allowed me to put “All of these” in the last slot so it was a constant. This is a feature they call “pinning”.

Getting the Results

The status of my survey was a bit confusing for me. In looking at the dashboard, it appeared that it ran two days, from March 19th through the 20th, to get my 200 respondents.

Dashboard show survey times
Dashboard Showing Survey Status

While it took 2 days to collect the data, it seems there is an additional analysis phase. I didn’t get my “survey complete” email till March 30th. The other part that mystifies me is if the survey ran two days, why did I see results in the “Responses by day of week” section for other days? I expected just to see bars for Tuesday and Wednesday.

email indicating survey completion
Survey Completion Email

My survey was small with 200 respondents and the majority selected “All of these” option for my second question. It also looks like some people who responded affirmatively to Question 1, didn’t answer Question 2. If I were to run this again, I would choose a different question type.

results for question 2
Results For Question 2

Drilling Down

One feature that I think would be very useful for people that run larger surveys is that you can drill-down into the data. I could click an item from the left side to see how a specific segment answered. The survey system infers certain information about the respondent based on a DoubleClick cookie, IP location and their Google Ad settings.

survey graphs are interactive
Interactive Survey Results

Another feature of the service is it flags “insights”. This is where Google detected a pattern about the survey responders. Even though I had a tiny survey, the system spotted 2 insights out of a possible 60. The number of possibilities varies based on number of questions. You can get a better idea of these insights by looking at some of the example surveys such as this one on dog toys.

example of survey insights
Survey Insights Example

Without Google’s coupon, I probably wouldn’t have tried the service as soon as I did. I clearly made mistakes in setting up my survey, but I did come away with a better idea of how the service works. In retrospect, I should’ve read the documentation better and not rushed in. However, I think there’s an opportunity for Google to improve their online help. They could do a better job of teaching people how to use the service and why certain survey question formats work better than others.

I’ll most likely use Google Consumer Surveys again. I think this format makes it easier to test markets and assumptions at an economical price. I’ll just spend longer designing the service. And who knows, maybe I’ll also test it out as a publisher on another site to see how it works for monetization. My final suggestion would be to start with a small survey to get acquainted with the service. I’m sure if you poke around, you’ll see a coupon offer appear.

Written by: TD
Google Consumer Surveys
Date published: 04/15/2013
4 / 5 stars