What is the Problem with Surveys?

What is Going on in My Head?

As a small business owner, wouldn't it be a dream come true to get inside the heads of your target audience? To intimately understand their likes, dislikes, concerns, questions and problems. To feel their desires, pleasure and pain. To hear the conversion conversation going on inside their head as it relates to their problem and what they perceive the solution to look like. To see with their eyes and life experience.

What a competitive advantage that would be. We all know what it feels like to be a customer but for some odd reason, it is extremely difficult to maintain the connection when we jump the fence to the selling side. This is especially true online as we rarely come face-to-face with our potential customers.

One way to get insider information is to have your customer service representatives actively listen and ask questions when conversing with customers. Obviously this requires extra effort and may fall through the cracks when your CSRs get slammed with orders.

Maybe you sell electronic scooters (the kind for kids) on your website but your conversion rate is really low. You tweak your copy, lower your prices, offer free shipping, etc… Still no change and you wonder what your visitors want. You do some research and notice a few companies selling mobility scooters (the kind for adults). You create a landing page split testing two options/paths; one for electronic scooters and another for mobility scooters. You track visitor choices with Google Analytics and find out that 75% of all visitors searching for "electronic scooters" are actually interested in mobility scooters. You can either add mobility scooters to your product mix or stop targeting this keyword phrase. Either way, you now have accurate insider information.

If all of this sounds like more work, it is. Hence the reason most companies pass over these in favor of statistical surveys, opinion polls, quantitative marketing research, paid surveys, and questionnaires. One very popular and common online survey example is SurveyMonkey.com.

So, what is the problem with surveys you ask? Quite simply, you only get responses from those people who do surveys. Although you may have chosen a random sample, the data is biased by this fact alone. Beyond this, the data you collect depends on subjects' motivation, honesty, memory, and ability to respond. They may even be motivated to give answers that present themselves in a favorable light.

Survey question and answer choices could lead to vague data because they are relative to "strength of choice". The choice "moderately agree" may mean different things to different subjects, and to anyone interpreting the data for correlation. Even yes or no answers are problematic because subjects may put "no" if the choice "only once" is not available.

When I think back to all the surveys I have taken, I usually started off with good intentions. As I got further into the questioning, I often felt overwhelmed and unsure of how I felt. This led me to care less about how accurate my answers were and more about how much longer this was going to take.

I am no survey guru, but a quick search of Google will provide countless challenges and problems with surveys. I am simply not sold on the idea of surveys being the best way to get accurate information about your customers. Here is one terrific example of how surveys can fail and in a huge way:

The Infamous Literary Digest Poll and the Election of 1936

In 1936, Franklin Delano Roosevelt had been President of the United States for one term. The Literary Digest predicted Alf Landon would beat Roosevelt by 57 to 43 percent in that year's election. They mailed over 10 million questionnaires to names drawn primarily from automobile and telephone owner lists. Amazingly, over 2.3 million people responded thus creating a huge sample to work with even by today's standards.

A young man named George Gallup sampled only 50,000 people and predicted that FDR would win. He was ridiculed for being naive. The Digest had correctly predicted the winner in every election since 1916. They had based their latest predictions on the largest response to any poll in history. However, Roosevelt won with 62% of the vote. The size of the Digest's error was staggering. How could they have been so far off?

The Literary Digest made two fatal mistakes. The Great Depression began in 1929 and the underlying economic condition was still that of widespread and deep depression. First, their list was disproportionately wealthy and hence, disproportionately Republican. Only they could afford such luxuries as cars and telephones. Second, they depended on voluntary response. Roosevelt was the incumbent and those who were unhappy with his administration were more likely to respond to the survey.

When a sample is biased, a large number of subjects cannot correct for the error. Some think the wrong prediction was largely responsible for the magazine folding. Gallup remains highly respected for the quality of their work over 70 years later.

Be careful what you ask for because the answers depend on it!

Image Credit: Ian Muttoo