Data Collection Techniques and Instruments

Sometimes research is a matter of disciplined listening and watching. No real instrument will emerge in this kind of fundamental research efforts.  Most of the time - particularly when using survey or questionnaire methods, or attempting to evaluate a program or do content analysis on cultural artifacts or information systems, constructing a research instrument is a major part of the research process.  The term instrument refers to a question, a series of questions, an entire questionnaire or interview schedule, or rating system - the set of measuring tools to be used to make observations - and is the physical paper and ink set of items themselves. Just as an engineer might have sets of calipers, metal rules, or metered moisture detection equiment, the researcher in the behavioral sciences requires specific tools for measurement as well.

Issues of Reliability and Validity are important here and fundamental to the task of constructing a questionnaire. Servicability, or utility, and ease of implementation and scoring, and interpretation of responses are also important.  These are matters that are decided about during instrument construction and continue through pilot studies and "try-outs" or test runs.

Behavioral science has been around long enough that a number of resource books have been compiled containing thousands of tests and measures. Usually, in addition to verbatim reproductions of the measures themselves, such a resourced will sort measures by concept measured, and provide citations in the literature that detail the measure's performance record in terms of reliability and validity.

The use of most pre-existing measures are encouraged by the originator, even to the extent that the measure is often published with instructions for administration and coding. See this Resource Guide for Measurement. and take time to read and review this reading - An Indepth look at the Development of a Complex Concept Measure

Survey Research, using questionnaires or a schedule of interviewer-guided questions, is the most widely used data-gathering technique in all of social science because of convenience and economy. While costs of conducting survey research can skyrocket, it is the most economical form of social research for collecting data on hundreds of respondents at a time. Thus,  survey research also can be the most widely abused and mismanaged technique in all of social science. Several events, inventions, and technological advances have served to increase the viability of survey research:

• Sampling techniques have become very precise
• Development of measurements such as Scales and Indexes
• Development of statistical software to run on increasingly powerful desktop computers
• Advances in data storage
• Growth of academic consortia such as the National Opinion Research Center, and ISPCR
• Prolifereation of funding sources.

In other words, people who are heads of corporations, or who are running for re-election, or are otherwise in a position to suffer greatly if the rely on faulty data, are willing to trust survey research not to lead them down the wrong path, the 2004 Presidential exit polling debacle not withstanding.  Since the unit of analysis for questionnaires will always be the individual respondent, the survey method usually has the individual as its focus, although pairs or groups of individuals that form a logical study unit (a couple, a family, a company of workers) are often the actual unit under study.

1. Questions about behavior
    When did you last use our product?       How often to you use marijuana?    Have you ever had the opportunity to ...?
2. Attitudes/Beliefs/Opinions:
    Describe your level of religious belief.    Do you think people are basically good or bad?
    How well is the President doing on the following issues, in your opinion?
3. Characteristics:
    What is your age at your last birthday?    What is your height in feet and inches?    Do you own a handgun?
4. Expectations:
    In the next 12 months, do you expect any change in your financial standing?
    Do you think the problems in the Middle East will continue or begin to abate?
    If the Democratic party regains a majority in congress, will this result in good things for Americans?
5. Self-Classification:
    Are you a Liberal, a Conservative, or a moderate?    In which social class grouping would you say you belong to?
    Would you say you are more informed, or less informed than the average person?
6. Knowledge:
    Match the word on the right with the definition on the left.
    Is it your consitutional right to withhold your legal name for the police when asked?
    On which continent is the country of Lithuania?
7. Demographic measures:
    What is your gender?      What is your highest educational attainment?    What is your age?

1. Use of Jargon or specialized language - What is your opinion of HR 134.234?  Make sure your questionnaire is simple enough for every respondent to comprehend every word.
2. Ambiguity, confusion or vagueness.  Clear and concise is the way to go.
   
3. Don't be timid about asking for sensitive information, if your study requires it.  A question such as Have you ever had unprotected sex - that is, sexual contact with someone not your spouse and without using a condom? will likely terminate the interview with some respondents unless there is some kind of lead in or warning ... The next set of questions deals with some sensitive issues.  Please remember that your responses cannot be associated with you in any way, and that you may ask to move to the next question if you prefer not to answer. Then ask the questions and come to the point.  Place the most sensitive questions in the middle of the questionnaire.  If such information is vital to your study, think about introducing a rhythm of safe to sensitive to safe to sensitive questions. You want information and must maintain your resolve even though you may be a little embarrassed for your respondents. 
   
4. Emotional language and prestige bias. What is your opinion of political leaders who lie to their constituents to get young people to fight and die in needless wars? - No doubt about the personal sentiments of the author of this question.  Objectivity requires that the researcher tries to see all sides of every issue. Your concern in doing research is not to win converts, but to measure respondent beliefs and attitudes.  Such tactics in survey research lead to false conclusions. The question above is really two questions.  How do you believe the President was truthful with the people regarding the current war?  and  Do you believe the current war is justified, even if some American men and women may have to die?
5. Double-Barrelled Questions - Any question that subsumes more than one response, such as Do you like Peach and Cherry flavorings? Yes or No. Someone might like one and not the other and couldn't answer the question truthfully.
6. Leading Questions - such as, We all try to be nice to people, but sometimes people make us angry. When you shout at people, do you always start out trying to be nice?
7. Questions that are beyond respondent's capability to answer. Except in cases where the researcher is testing for respondent capabilities, asking questions outside the subject's knowledge is pointless and meaningless, and can fatique the respondent to the point of stopping the interview. There are exceptions to this rule, if the researcher is attempting to guage a respondent's knowledge or expertise in some conceptual area. Here the researcher would want to allow the respondent to continue to answer until they begin to fail.
   
8. False premises - like this: Clearly the conservative right is heading the country down the right path. In your opinion, how might we, as right-thinking Americans, head off this tide of greed? More than likely, the researcher will anger enough respondents to limit the validity of the dataset.
9. Future intentions - Be careful about asking respondents about their behavior in the future. Do you think you will buy a Chevy the next time you need a new car?  Instead, it is legitimate to ask about past behaviors and then to probe regarding whether or not the respondent would behave the same today, given some recent development.
   
10. Double negatives - Don't never use none of these type questions. Isn't that not confusing?
11. Overlapping or unbalanced response categories - Many times questionnaire items carry response categories that range evenly across all possibilities (i.e., Never Rarely Often Always). The idea is to avoid something like Never, Hardly Ever, Rarely, Once in a Blue Moon, Always.

Return to Syllabus