How to Design and Conduct Surveys

Copyright Carter McNamara, MBA, PhD; Authenticity Consulting, LLC

Also consider
Related Library Topics

Preparation

Before you design your questionnaire, clearly articulate what decision, problem or need is to be addressed by the answers to them. For example, review why you are doing the evaluation and what you hope to accomplish by it. This provides focus on what information you need and, ultimately, on what questions should be in the questionnaire. The following guidelines are relevant, whether you are using questionnaires on hardcopy or online.

Directions to Respondents

1. Include a brief explanation of the purpose of the questionnaire.
This also helps respondents to feel a sense of purpose and motivation for themselves when responding to the questionnaire.

2. Include clear explanation of how to complete the questionnaire.
It might be clear to you about how to do the questionnaire, but remember that others are seeing it for the first time. Unclear directions can produce unclear answers.

3. Include directions about where to provide the completed questionnaire.
For some questionnaires, it can take longer to provide them than to answer their questions. That can dissuade people from responding to it.

4. Note conditions of confidentiality.
Specify who will have access to the information and for what purposes. Explain how you will limit access and ensure confidentiality.

Content of Questions

1. Ask about what you need to know.
Get information about the goals or ultimate questions you want to address by the evaluation.

2. Ask questions that the respondent should be able to answer.
Make sure that the respondents can reasonably be expected to know the answers.

3. Ask questions that the respondents want to answer.
If the questions are too private or silly, the respondents may lose interest in the questionnaire.

Wording of Questions

1. Will the respondent understand the wording?
For example, are you using any slang, cultural-specific or technical words?

2. Are any words so strong that they might lead a respondent to a certain answer?
Attempt to avoid use of strong adjectives, for example, “highly effective government” or “prompt and reliable.”

3. Ensure you are asking one question at a time.
Try to avoid use of the word “and” in your question.

4. Avoid using “not” in your questions if you are asking yes/no questions.
Use of “not” can lead to double negatives and cause confusion.

5. In multiple-choice questions, be sure your choices are mutually exclusive.
Respondents should not be confused about whether alternatives mean the same thing.

Order of Questions

1. Engage respondents early in questionnaire to increase motivation to complete it.
Start with fact-based questions and then go on to opinion-based questions. For example, ask people for demographic information about themselves and then go on to questions about their opinions and perspectives. This helps respondents to feel confident and relaxed before encountering more challenging and reflective questions about their opinions.

2. Attempt to get respondents’ commentary in addition to their ratings.
For example, if the questionnaire asks respondents to choose an answer by circling an answer or provide a rating, then perhaps ask them to also provide commentary that explains their choices. Be sure to provide sufficient space and time to include commentary.

3. Include a question to get respondents’ impressions of the questionnaire itself.
For example, ask them if the questionnaire was straightforward to complete (“yes” or “no”), and if not, to provide suggestions about how to improve the questionnaire. This type of question can be extremely useful when modifying or designing questionnaires.

4. Test your questionnaire on a small group of people before including more people. Ask them if the questions seemed straightforward. Carefully review their answers. Do you understand them? Do their answers ultimately help you to answer your research questions? What changes should you make to the questionnaire?

5. Finish the questionnaire. Finish the questionnaire according to results of the pilot. Mark the questionnaire as being a master or baseline document. Put a date on it so you can keep track of all future versions.

Various Perspectives on Conducting Surveys

• Sampling and surveying handbook
• Guidelines for Employee Opinion Surveys
• Survey Research – A Basic Primer
• SPSS – surveys, sampling, etc.
• Resources for survey researchers
• Business Survey Methods
• Guide to Designing Online Surveys (uses compensation survey as example)
• The Secret Life of Surveys
• How to Design an Employee Attitude Survey
• Keeping on Top of Talent With Online Surveys
• Lacie Barber on Survey Methodology Tips for Improving Quantitative Data Quality
• Employee Surveys: If You Measure It, You Manage It
• Basics of Satisfaction Survey Design – Part 1 of 4
• Basics of Satisfaction Survey Design – Part 2 of 4
• Basics of Satisfaction Survey Design – Part 3 of 4
• Basics of Satisfaction Survey Design – Part 4 of 4
• The Best Satisfaction Surveys
• How to Simplify Survey Design

---

For the Category of Communications (Business Writing):

To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.

• Related Library Topics
• Recommended Books

Selecting Which Business Research Method to Use

© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.
Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation and Field Guide to Consulting and Organizational Development.

Sections of This Topic Include

• Overview of Research Methods
• Four Levels of Research Results

Also consider
Related Library Topics

Learn More in the Library’s Blogs Related to Selecting Research Methods

In addition to the information on this current page, see the following blogs which have posts related to Selecting Research Methods. Scan down the blog’s page to see various posts. Also see the section “Recent Blog Posts” in the sidebar of the blog or click on “next” near the bottom of a post in the blog.

• Library’s Business Planning Blog
• Library’s Strategic Planning Blog

---

Before selecting the methods you will use in your research, be sure that you have read the topic Planning
Your Research.

Overview of Research Methods

The following table provides an overview of the major methods used for collecting data during evaluations.

Method	Overall Purpose	Advantages	Challenges
questionnaires, surveys, checklists	when need to quickly and/or easily get lots of information from people in a non threatening way	-can complete anonymously -inexpensive to administer -easy to compare and analyze -administer to many people -can get lots of data -many sample questionnaires already exist	-might not get careful feedback -wording can bias client’s responses -are impersonal -in surveys, may need sampling expert – doesn’t get full story
interviews	when want to fully understand someone’s impressions or experiences, or learn more about their answers to questionnaires	-get full range and depth of information -develops relationship with client -can be flexible with client	-can take much time -can be hard to analyze and compare -can be costly -interviewer can bias client’s responses
documentation review	when want impression of how program operates without interrupting the program; is from review of applications, finances, memos, minutes, etc.	-get comprehensive and historical information -doesn’t interrupt program or client’s routine in program -information already exists -few biases about information	-often takes much time -info may be incomplete -need to be quite clear about what looking for -not flexible means to get data; data restricted to what already exists
observation	to gather accurate information about how a program actually operates, particularly about processes	-view operations of a program as they are actually occurring -can adapt to events as they occur	-can be difficult to interpret seen behaviors -can be complex to categorize observations -can influence behaviors of program participants -can be expensive
focus groups	explore a topic in depth through group discussion, e.g., about reactions to an experience or suggestion, understanding common complaints, etc.; useful in evaluation and marketing	-quickly and reliably get common impressions -can be efficient way to get much range and depth of information in short time – can convey key information about programs	-can be hard to analyze responses -need good facilitator for safety and closure -difficult to schedule 6-8 people together
case studies	to fully understand or depict client’s experiences in a program, and conduct comprehensive examination through cross comparison of cases	-fully depicts client’s experience in program input, process and results -powerful means to portray program to outsiders	-usually quite time consuming to collect, organize and describe -represents depth of information, rather than breadth

Also consider:

• Appreciative Inquiry
• Survey Design

Four Levels of Research Results

There are four levels of information that can be gathered from customers or clients, including getting their:
1. reactions and feelings (feelings are often poor indicators that your service made lasting impact)
2. learning (enhanced attitudes, perceptions or knowledge)
3. changes in skills (applied the learning to enhance behaviors)
4. effectiveness (improved performance because of enhanced behaviors)

Usually, the farther your research results get down the list, the more useful is your research results. Unfortunately, it is quite difficult to reliably get information about effectiveness. Still, information about learning and skills is quite useful.

---

For the Category of Business Research:

To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.

• Related Library Topics
• Recommended Books

Sample Case Study Report

© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.

(This document is referenced from Case Study Design.)

Here is a sample of a case study report. Evaluation included focus on the program’s process, outcomes and facilitation. The following case study was used to convey the funder, a holistic depiction of the experience and outcomes from the program. “Jack” is a fictional name in this sample.

“Jack”
Jack is a chief executive of an organization with a small budget and staff. Jack’s overall goal in his circle was to communicate with other executives about projects and challenges they face, including brainstorming solutions together. He mentioned numerous challenges that he faced in running his organization, some of which needed specific, technical information to address.

In the first meeting, he mentioned other issues that he wanted to address. In comparison to other members in his circle, he had the most issues. He also wanted help managing his time more effectively and he wanted to improve the effectiveness of his board. He stated, “They don’t even know what we’re about. They just sit there when we meet.” And he wanted to improve his understanding of his role as a chief executive. He asked, “How do I know what I can ask the board to do?” “What is my role with them?” In another area, he said he wanted some ideas about how to expand his organization’s revenue.

In the first meeting, members asked him many questions, mostly to obtain additional information about
his issues. Jack responded that his most pressing project was time management. Another member responded, “I’d challenge you on that,” and asked Jack if he would have more time if he got more support from his board. Jack laughed and answered, “I suppose so.” Other group members concurred. From his first meeting, he took away actions including listing and ranking his issues, scheduling a time management course, and identifying a course that would provide an overview of the chief executive’s role. One member asked him to also list and rank his issues for the next meeting.

In the second meeting, Jack produced the following list: improving his understanding of the chief executive and board roles, developing/energizing the board, and conducting strategic planning with the board that would include expanding the revenue in his organization. Other group members agreed with Jack’s list.

In discussion in the second meeting, Jack acknowledged that he was doing more as an chief executive than is usually expected from that role. He also realized that he was overloaded because he got little or no support from his board. He indicated that he did not feel confident, though, approaching his board members for more support. As a result of other circle members’ support and coaching, he resolved to approach the board — and a month later, he had. He and the board members committed to complete board training. He arranged training to include strong focus on strategic planning, which included expanding revenue. To further build rapport, he elected to have lunch with one board member a month, including giving them a tour of the organization.

Jack noted on his evaluation questionnaire: “[The process had!!] just the right amount of structure. The conversation is pretty free-flowing, but there’s enough attention paid to time so that everyone gets a fair chance.” His top reported outcomes were in the categories of access to a network, professional development, and effectiveness. “[The program provided me!!] an opportunity to meet other chief executives and hear about projects that they faced and how they handled those projects.” He stated, “The program has restored some order to my job,” and “A lot has happened with my job.”

---

For the Category of Evaluations (Many Kinds):

To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.

• Related Library Topics
• Recommended Books

Basics of Developing Questionnaires

© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.

Adapted from the Field Guide to Consulting and Organizational Development.

Whether developing questions for questionnaires or interviews or focus groups, there are certain guidelines that help to ensure that respondents provide information that is useful and can later be analyzed.

Sections of This Topic Include

• Types of Information Collected by Questions
• Two Types of Questions
• Key Preparation
• Directions to Respondents
• Content of the Questions
• Wording of the Questions
• Order of the Questions

Also consider

• Related Library Topics
• There Is No Hope Of Doing Perfect Research
• Creating and Implementing Your Data Collection Plan

Learn More in the Library’s Blogs Related to Developing Questionnaires

In addition to the articles on this current page, also see the following blogs that have posts related to Developing Questionnaires. Scan down the blog’s page to see various posts. Also see the section “Recent Blog Posts” in the sidebar of the blog or click on “next” near the bottom of a post in the blog. The blog also links to numerous free related resources.

• Library’s Business Planning Blog
• Library’s Building a Business Blog
• Library’s Strategic Planning Blog

---

Types of Information Collected by Questions

Questions are geared to find out what people know, did, feel and think.
1. To find out what information they know, ask them to describe something, e.g., “Please describe …”
2. To find out what they feel, ask them, e.g., “How do you feel about …?” or “How did you feel when …?”
3. To find out what they think, ask them for their opinion on something, e.g., “How could the .. be improved?”
4. To find out what they did, ask them to describe an activity they did.

Two Types of Questions

1. Open-ended:
No options are provided for the respondent to answer the question. They must think of their own response and describe it in their own words. If respondents have and take the time to reflect on answers to the question, you can get more meaningful information than from closed questions.

2. Closed:
The respondent is given a set of alternative choices from which he or she can choose to answer the question, i.e., “yes,” “no,” multiple choice, a rating, ranking, etc. Closed questions can usually be answered quickly, allowing you to get a get a lot of information quickly. However, respondents may rush through the questions and not take enough time to think about their answers. Your choices may not include the answer they prefer.

How you configure your questions together, depends on whether they’re used in questionnaires, interviews or focus groups.

Key Preparation

Before you start to design your questions, clearly articulate what problem or need is to be addressed using the information to be gathered by the questions. Review why you’re doing the evaluation and what you hope to accomplish by it. This provides focus on what information you need and, ultimately, on what questions should be used. (See Planning Your Program Evaluation.)

Directions to Respondents

1. Include a brief explanation of the purpose of the questionnaire.
2. Include clear explanation of how to complete the questionnaire.
3. Include directions about where to provide the completed questionnaire.
4. Note conditions of confidentiality, e.g., who will have access to the information, if you’re going to attempt to keep their answers private and only accessed by yourself and/or someone who will collate answers. (Note that you not guarantee confidentiality about their answers. If a court sued to see answers, you would not likely be able to stop access to this information. However, you can assure that you will make every reasonable attempt to protect access to their answers. You should consider using an informed consent form, as well.)

Content of Questions

1. Ask about what you need to know, i.e., get information in regard to the goals or ultimate questions you want to address by the evaluation.
2. Will the respondent be able to answer your question, i.e., do they know the answer?
3. Will respondents want to answer the question, i.e., is it too private or silly?

Wording of Questions

1. Will the respondent understand the wording, i.e., are you using any slang, cultural-specific or technical words?
2. Are any words so strong that they might influence the respondent to answer a certain way? Attempt to avoid use of strong adjectives with nouns in the questions, e.g., “highly effective government,” “prompt and reliable,” etc.
3. To ensure you’re asking one question at a time, avoid use of the word “and” in your question.
4. Avoid using “not” in your questions if you’re having respondents answer “yes” or “no” to a question. Use of “not” can lead to double negatives, and cause confusion.
5. If you use multiple choice questions, be sure your choices are mutually exclusive and encompass the total range of answers. Respondents should not be confused about whether two or more alternatives appear to mean the same thing. Respondents also should not have a clearly preferred answer that is not among the alternative choices of an answer to the question.

Order of Questions

1. Be careful not to include so many questions that potential respondents are dissuaded from responding.
2. Attempt to get recruit respondents’ motivation to complete the questionnaire. Start with fact-based questions and then go on to opinion-based questions, e.g., ask people for demographic information about themselves and then go on to questions about their opinions and perspectives. This gets respondents engaged in the questionnaire and warmed up before more challenging and reflective questions about their opinions. (Consider if they can complete the questionnaire anonymously; if so, indicate this on the form where you ask for their name.)
3. Attempt to get respondents’ commentary in addition to their ratings, e.g., if the questionnaire ask respondents to choose an answer by circling an answer or provide a rating, ask them to provide commentary that explains their choices.
4. Include a question to get respondents’ impressions of the questionnaire itself. For example, ask them if the questionnaire was straightforward to complete (“yes” or “no), and if not, to provide suggestions about how to improve the questionnaire.
5. Pilot or test your questionnaire on a small group of clients or fellow staff. Ask them if the form and questions seemed straightforward. Carefully review the answers on the questionnaires. Does the information answer the evaluation questions or provide what you want to know about the program or its specific services? What else would you like to know?
6. Finalize the questionnaire. Finalize the questionnaire according to results of the pilot. Put a date on the form so you can keep track of all future versions.

---

For the Category of Evaluations (Many Kinds):

To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.

• Related Library Topics
• Recommended Books

Planning Your Business Research

© Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC.
A Field Guide to Nonprofit Program Design, Marketing and Evaluation

Sections of This Topic Include

• Research Plans Depend on Information You Need and Available Resources
• Key Considerations to Design Your Research Approach

Also consider

• Related Library Topics
• There Is No Hope Of Doing Perfect Research
• Creating and Implementing Your Data Collection Plan

---

Description

The following information is intended to give the reader some general guidance about planning a basic research effort in their organization. The rest of the information in the section presents an overview of methods used in business, how to apply them, and how to analyze and interpret and report results.

Research Plans Depend on Information You Need and Available Resources

Often, organization members want to know everything about their products, services, programs, etc. Your research plans depend on what information you need to collect in order to make major decisions about a product, service, program, etc. Usually, you’re faced with a major decision due to, e.g., ongoing complaints from customers, need to convince funders / bankers to loan money, unmet needs among customers, the need to polish an internal process, etc.

The more focused you are about what you want to gain by your research, the more effective and efficient you can be in your research, the shorter the time it will take you and ultimately the less it will cost you (whether in your own time, the time of your employees and/or the time of a consultant).

There are trade offs, too, in the breadth and depth of information you get. The more breadth you want, usually the less depth you’ll get (unless you have a great deal of resources to carry out the research). On the other hand, if you want to examine a certain aspect of a product, service, program, eta., in great detail, you will likely not get as much information about other aspects as well.

For those starting out in research or who have very limited resources, they can use various methods to get a good mix of breadth and depth of information. They can understand more about certain areas of heir products, services, programs, eta., and not go bankrupt doing so.

Key Considerations to Design Your Research Approach

Good business research is about collecting the information you really need, when you need it, to answer important questions and make important business decisions. What is the key to doing good business research? To make the best use of your time, get the information you really need, and make the best business decision, consider the following key questions before doing your research:

1. Why am I doing this research? What important decision am I trying to make?

Always have an important decision in mind when you are doing your research. You are too busy to waste time collecting information to help make a decision that is not vital to your business, or worse yet – collecting information with no purpose in mind. With a clear decision in mind, you will be able to keep your research focused.

2. When do I need to make my decision?

Timing is everything in business. Having 60% of the questions answered in time to make your decision is better than having 100% of the answers after the deadline’s passed. But on the other hand, if your important decision really can wait, there’s no sense in rushing into things and acting on less information that you might have been able to get if you had taken your time. So you need to have a clear sense of when you need to make your important decision.

3. What questions do I really need to answer to make my decision? What information do I really need to answer my questions?

This is where many people get lost in their research. What do you really need to know to be able to make your business decision? Do you need to know a little about a bunch of things, or a lot about a few things? What kind of information do you need? Numbers? Opinions? And how much is enough? (A good rule of thumb is, the more important the decision, the better the information you should collect.) How you answer these questions will have a big impact on where you are going to have to go to get your information, and how you are going to get it.

4. Where is the best place (and who are the best people) to get the information I really need?

Overall, information sources can be broken down into two kinds: primary and secondary. Primary sources are those people and organizations in your marketplace, for example, your potential customers, suppliers, and competitors. Secondary sources are reports, articles, and statistics about the people in your marketplace.

While there are exceptions, it is usually safe to start with your secondary sources, because the information’s usually readily available at low or no cost. Once you have gotten what you can from the secondary sources, ask yourself the question, “Do I really need more information to make my decision?” If you really do, turn your attention to your primary information sources to get the last vital pieces of information you need. But often you can get what you really need from secondary sources.

The real challenge for you with secondary information sources is not having too little information. You will likely be faced with a large amount of information for any decision. The real challenge will be to selectively pick the best from what is available. And it is always a good idea to use at least two good sources of information for any decision, and to make sure that these different sources agree with each other.

If you have done things right up to this point, selecting your sources – primary and secondary – should  not be too hard. You will know what decision you are trying to make and when you need to make it, and you will know what information you really need to make that decision. And if you can explain this to the reference librarian at your local library, they will get you pointed in the right direction. It is worth noting that many people go “researching” way before they really know what they are researching – and they waste a lot of time in the process.

5. What options do I have to collect that information?

With secondary information sources, collection is straightforward. You go to the source (library, resource center or website) and ask for the information. With primary information sources, deciding upon the right method is a little more involved. When considering your options, always remember to keep your business decision, timing and the information you really need clearly in your mind. These will help you to make the best decision.

6. What resources do I have to collect that information? Who or what can help me?

You are almost ready to go out and do your research. One final consideration is about the resources you have, or have access to. These resources can include:

• The time you are willing to commit
• Friends and family members who are willing and able to help you
• The money you are willing and able to spend
• Access to the internet, your trainer
• Other resource people in your community like the reference librarian at your local library

7. Given the time, options, and resources I have, what is the best way for me to get the information I need?

Now it is time to make a decision about how you are going to do your research. This is not so much a separate step as it is something that will emerge as you go through the earlier steps. Still, it is good to stop and think it through one last time before you move forward.

8. What am I actually going to do and when?

Okay – it is time to commit to a plan of action. Create a business research action plan to collect your thoughts.

Now you are ready to consider various methods to implement your plan. See Business Research Methods.

9. What ethical considerations might there be in collecting information?

For example, will any of the participants be quoted in research reports? If so, then you should get their explicit information to do that.
Ethics and Conducting Research

---

Learn More in the Library’s Blogs Related to Planning Business Research

In addition to the articles on this current page, also see the following blogs that have posts related to Planning Business Research. Scan down the blog’s page to see various posts. Also see the section “Recent Blog Posts” in the sidebar of the blog or click on “next” near the bottom of a post in the blog. The blog also links to numerous free related resources.

• Library’s Business Planning Blog
• Library’s Building a Business Blog
• Library’s Strategic Planning Blog

---

For the Category of Business Research:

To round out your knowledge of this Library topic, you may want to review some related topics, available from the link below. Each of the related topics includes free, online resources.

Also, scan the Recommended Books listed below. They have been selected for their relevance and highly practical nature.

• Related Library Topics
• Recommended Books