Month: March 2012

Posted on

A Guide to Navigating the Evaluation Maze: “A Framework for Evaluation” from the Centers for Disease Control and Prevention (CDC), Part 2

A-man-thinking-of-how-to-work-through-a-maze

This is part 2 of a previous post on the Centers for Disease Control and Prevention’s (CDC) evaluation model. The goal of these posts is not to give an exhaustive description of this model but to whet your appetite for further study, to refer you to other sources and to share with you some related topics that have been percolating in my head.

In the last post, we covered steps 1-3 of the CDC’s evaluation depicted below:

A Framework for Evaluation.

Source: Centers for Disease Control and Prevention (CDC), Office of the Associate Director for Program (OADPG)

Step 4: Gather Credible Evidence

What is credible evidence? Let us back up and consider credibility from various perspectives–those of funders, agency staff and program participants. Involving most important stakeholder groups throughout the evaluation process and being open to learn from their experiences will increase the credibility of the evidence. Some program participants may not believe sources like government agencies and doctors who are traditionally viewed as credible sources in professional circles.

Your evidence is only as good as the tools you use to collect it. Use high quality tools, i.e., questionnaires, interview guides, etc. Pay attention to validity issues–for e.g. do the questions really measure what you think they are measuring? At the very least, choose indicators based on a review of the literature. Indicators are items being measured such as knowledge levels, numbers of low birth weights, etc. that shed light on the health or social condition that your program is trying to change.

Ask experts to review your evaluation tools and then pilot test them among program participants. In some cases, it may be particularly important to use an evaluation tool that has been tested for reliability, i.e., does the questionnaire yield consistent results each time it is used? In these cases I recommend, if possible, using a tool that has been published in the peer-reviewed literature. University libraries often allow visitors to use their databases and to access peer-reviewed journals online. Other tips from the CDC–the CDC recommends:

  • choosing indicators wisely
  • training staff in data collection,
  • paying attention to data quality issues and
  • protecting confidentiality of participants’ information

Step 5: Justify Conclusions

All conclusions need to be based on evidence. Take care also to base all your conclusions on sound statistical reasoning. For e.g., one common mistake is to conclude that there is a cause and effect relationship on the basis of correlational data. A statistical correlation only shows that two variables are associated with one another. Take for example the following piece of evidence: depression is correlated with lower levels of perceived social support. All we can conclude is that there is a correlation between depression and social support. Lower levels of perceived social support could have contributed to the depression, or the depression itself could have led to social withdrawal, which then resulted in lower levels of perceived social support. If you’re interested in a light and amusing read to familiarize yourself with such principles, I second evaluator John Gargani’s recommendation of Darrell Huff’s classic book “How to Lie with Statistics.”

This is another step where it is important to continue engaging stakeholders. Encourage stakeholders to participate in the process of drawing conclusions from evidence. This will increase their trust in the findings and will increase the chances that they will actually use the evaluation.

Step 6: Ensure Use and Share Lessons Learned

So how do we ensure that evaluation findings are actually used? Like cooking, presentation is everything! People process visual information much more intuitively and naturally than verbal information. Consider, for example, how well very young children respond to colors and pictures. This principle translates into effectively communicating your findings to adult audiences as well. A hot topic in the field of evaluation is data visualization or how to display information using sound design principles. While it is true that graphs can be confusing, effectively applying data visualization principles can produce graphs that are elegantly intuitive to a lay evaluation consumer. For further study, read Edward Tufte’s classic book “The Visual Display of Quantitative Information.”

A tool that helps visually depict a variety of graphs and charts in one place is a data dashboard. Think of it as a short cut to communicating information visually. A dashboard is a display of multiple graphs all in the same location. A resource for further reference is Stephen Few’s Information Dashboard Design: The Effective Visual Communication of Data.

To increase use of the evaluation findings, the CDC recommends:

  • aligning evaluation design with how stakeholders plan to use the evaluation,
  • translating findings into practical recommendations for the program and
  • using reporting strategies that are customized to stakeholders’ specific needs.

For DIY (Do It Yourself) evaluators, I highly recommend renowned evaluation theorist Michael Quinn Patton’s book Utilization-Focused Evaluation. You can listen to free recordings of two webinars by Michael Q. Patton here. (You may need to download software first). “But I’m too busy managing my program to sit down and listen to webinars,” you protest.

I understand too well! Well, I listened to both webinars recently while doing housework and received very helpful guidance for a current project.

(To be continued)

——————

For more resources, see our Library topic Nonprofit Capacity Building.

____________________________________________________________________________________________________

Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation. Contact her at priyasusansmall@gmail.com. Visit her website at http://www.priyasmall.wordpress.com. See her profile at http://www.linkedin.com/in/priyasmall/

Posted on

Basics of Satisfaction Survey Design – Part 3 of 4

A feedback survey response on a desk

Common Survey Question Types and Examples

Multiple choice questions

Multiple choice questions have two or more answer options. Useful for all types of feedback, including collecting demographic information. Answers can be “yes/no” or a choice of multiple answers. Beware of leaving out an answer option, or using answer options that are not mutually exclusive.

Example 1: Are you a U.S. Citizen? Yes / No

Example 2: How many times have you called our agency about this issue in the past month?

□ Once

□ Twice

□ Three times

□ More than three times

□ Don’t know/not sure

Rank order scale questions

Questions that require the ranking of potential answer choices by a specific characteristic. These questions can provide insight into how important something is to a customer. Best in online or paper surveys, but doesn’t work too well in phone surveys.

Example: Please rank the following customer service factors, from most to least important to you, when interacting with us. (1=most important; 5=least important)

1st 2nd 3rd 4th 5th

□ □ □ □ □ Call wait time

□ □ □ □ □ Call hold time

□ □ □ □ □ Representative’s customer service skills

□ □ □ □ □ Representative’s knowledge skills

□ □ □ □ □ Resolution of issue

Rating scale questions

Rating scale questions that use a rating scale for responses. This type of question is useful for determining the prevalence of an attitude, opinion, knowledge or behavior.

There are two common types of scales:

1. Likert scale

Participants are typically asked whether they agree or disagree with a statement. Responses often range from “strongly disagree” to “strongly agree,” with five total answer options. (For additional answer options, see table below.)

Each option is ascribed a score or weight (1 = strong disagree to 5 = strongly agree), and these scores can be used in survey response analysis. For scaled questions, it is important to include a “neutral” category (“Neither Agree nor Disagree” below).

Example: “The customer service representative was knowledgeable”

□ 1. Strongly Disagree

□ 2. Disagree

□ 3. Neither Agree nor Disagree

□ 4. Agree

□ 5. Strongly Agree

2. Semantic differential scale

In a question using a semantic differential scale, the ends of the scale are labeled with contrasting statements. The scales can vary, typically using either five or seven points.

Example: “How would you describe your experience navigating our website?

□ 1. Very Hard to Navigate

□ 2. Somewhat Hard

□ 3. Neither Hard nor Easy

□ 4. Somewhat Easy

□ 5. Very Easy to Navigate

Open-ended questions

Open-ended questions have no specified answer choices. These are particularly helpful for collecting feedback from your participants about their attitudes or opinions. However, these questions may require extra time or can be challenging to answer, so participants may skip the questions or abandon the survey.

In addition, the analysis of open-ended questions can be difficult to automate, and may require extra time or resources to review. Consider providing extra motivation to elicit a response (e.g., “Your comments will help us improve our website”) and ensure there is enough space for a complete response.

Example: What are two ways we could have improved your experience with our agency today? We take your feedback very seriously and review comments daily.

Next time, we’ll discuss common mistakes and how to avoid them

(Many thanks to USA.gov for guidance on question design.)

——————

For more resources, see our Library topics Marketing and Social Networking.

.. _____ ..

ABOUT Lisa M. Chapman:

Ms. Chapman’s new book has a name change! The Net Powered Entrepreneur – A Step-by-Step Guide will be available in April 2012. Lisa M. Chapman serves her clients as a business and marketing coach, business planning consultant and social media consultant. She helps clients to establish and enhance their online brand, attract their target market, engage them in meaningful social media conversations, and convert online traffic into revenues. Email: Lisa @ LisaChapman.com

Posted on

The CFC: Choosing Your Leadership Development Team

A team working to achieve development goals

It’s almost spring, and now’s the time to start planning your non-profit’s CFC campaign for the 2012 campaign season.

The best-selling author, Jim Collins, who wrote Good to Great, has five key principles for organizational success; and, in his own words, the single most important principle is to “Get the right people on the bus.”

That means that selecting the right people for your organization or project team is one of the most important tasks of a leader. This, most certainly, applies to your CFC action team, whether they are all paid staff or some combination of paid staff and volunteers.

There are several factors to consider when forming your CFC action team, and in particular deciding who is going to become the project leader. One key question I have my consulting clients’ executive directors ask themselves, as they consider who should be the CFC Team Leader, is:
    “If I need to send one person to a CFC charity fair, who do I
    want representing our non-profit and its mission to the public?”

In addition to deciding upon a team leader for your non-profit’s CFC program, plan on having at least two others involved on a regular basis (and more if warranted):

  • A second person that can also staff the CFC special events including campaign kick-offs and charity fairs.
  • Someone who can answer basic questions about your non-profit’s participation in the CFC, while others are away from the office.
  • Depending up on the nature of your non-profit, it can be very valuable to have someone is part of the program staff as one of the key members of the CFC action team.

While it’s important to have at least one person who is in the office on a regular basis as the lead on your CFC action team, it’s often quite successful to have volunteers who are able to staff the charity fairs as well.

A few years ago the Chesapeake Bay Foundation had one volunteer whose sole activity was to staff charity fairs. As she once quipped to me, “I love helping the environment, but I don’t like getting muddy!”

Planning & Organizing

The team leader will be responsible for the overall planning for the non-profit’s CFC campaign, including deciding what campaign giveaways should be ordered (described in the previous post), as well as planning the communication and awareness generation strategies for the entire year, not just during the solicitation period.

The planning process include both deciding what tasks need to be accomplished, by what date, as well as a review of what worked well in the 2011 campaign (for non-profits already in the CFC), and what areas need improvement or updating for the 2012 campaign.

One item that needs to be periodically updated is your non-profit’s display board that has highlights about what your organization accomplishes. If the pictures haven’t been updated in several years, it’s time!

Annual Reports

Of the 7 Keys to CFC Success, the most important one is Number Seven, “Say Thank You Early and Often!” And, one of the places where you should say thank you is in your printed materials, including annual reports and newsletters.

Since many non-profits are now working on the content of their 2011 annual report, make sure that you include at least a simple thank you to both your CFC donors and to the CFC volunteers who helped raise funds for all of the CFC charities.

In the next post, we’ll take a look at one specific action that you can take in April to increase the awareness of your non-profit in your community.

=-=-=-=-=-=-=-=-=-=-=-=-=-=

During his 25-year career in the Federal sector, Bill Huddleston, The CFC Coach, served in many CFC roles. If you want to participate in the Combined Federal Campaign, maximize your nonprofit’s CFC revenues, or just ask a few questions, contact … Bill Huddleston
=-=-=-=-=-=-=-=-=-=-=-=-=-=

If you would like to comment/expand on the above, or would just like to offer your thoughts on the subject of this posting, we encourage you to “Leave a Reply” at the bottom of this page, click on the feedback link at the top of the page, or send an email to the author of this posting.

Posted on

All That You Think, Say and Do

A-man-thinking-of-how-how-to-do-project.

I love the music and lyrics of Faith Rivera. She’s a fabulous singer and messenger. Her performances are energizing, inspiring, and touching. I want to share this video clip that I use in some of my programs to get energized.

Listen to the music and let it fill you with positive energy. See which lines you like. My favorites are:

  • All that you think, say, and do, shapes the world with your truth
  • If you come from need, need will be the seed of your story
  • Tomorrow you can choose again

Enjoy!

Let me know how you like this video by sharing your reflections and comments below.

*******************

For more resources, see our Library topic Spirituality in the Workplace.

——————

Visit Linda’s website- www.lindajferguson.com for information about her coaching work, keynote presentations, seminar topics, and books.

Now available!! Linda’s new book, “Staying Grounded in Shifting Sand” – Click here to order.

“Friend” Linda on her Facebook page:

https://www.facebook.com/lindafergconsciouscreator

Click this link to order Linda’s 10th Anniversary edition of “Path for Greatness: Work as Spiritual Service”.

Posted on

Poorest Customer Service in the Land Where it Really Counts

A person opening the customer care interface with a laptop

Banking customer service start off on the phone with a dirge of automatic services to frustrate you long enough to want to hang up and start all over again, this time with a live person asking the same questions.

No, it’s not a Fairy Tale. It’s more like a rave about bad customer service, why they don’t give good customer service, and naturally, why we should, which gets us to the training part of this. I guess I should feel lucky to not live in a third-world county, but I think I’d get a straight answer there.

Ever notice, aside from Fast Food Joints (except Chik-Fila), the worst customer service folks seem to work for doctors, those very people who should be caring about how we are when we walk in the door, stupidly or not, thinking this is a place to get immediate medical help. Do it by phone, it’s even worse. If it’s an emergency call, 911. Really?

My emergency has a to do with a question that has to do with your “doctor” speciality. Granted some specialists are great. Those that deal with Cancer victims, “we jump for them” but still make them fill out the forms and aren’t terribly concerned (unless they ask) if they are able, so weak and trembling the daily treatment we are giving them, but hey, we do remember their names seeing them every day. Nice touch but not enough.

It seems the lower on the specialty scale the customers can even degrade to rudeness and then when the following Monday someone from the actual office calls you back and chews you out for giving the step-in who did call you for not giving the right number. You were told once before or it might be in multitudes of handouts after the multitudes of legal forms you filled out to make sure you will pay your bill even if insurance won’t. Business chewing client out. If I die because I didn’t, I’m sure there is a paper for that.

Besides banks, it appears doctor’s offices rank right up there with the only business who insurance companies can insist on you working with, depending on how much money you have. You want the best, pay for it. Bet they have good customer service.

Banking customer service start off on the phone with a dirge of automatic services to frustrate you long enough to want to hang up and start all over again, this time with a live person asking the same questions. My wife and I found unwarranted charges.

We spent nothing on the 5th of Whatever; my wife was getting Cancer Treatment all day. We never buy these product lines that showed on our statement. Clearly, not just flag raisers but flag wavers. Some one should have called us before our checking and Debit account was totally wiped out.

Meanwhile it is up to us to use the no-so-friendly tools listed on their so-so friendly website to find help and assurance this won’t happen again. You see if we had been smart and spent a lot of time looking at every inch of their website it wouldn’t have happened in the first place. Of course, we don’t know any banks or financial institutions that needed bailing out. Now I need bailing out.

My other credit cards should hold out long enough to make co-pays, but not the three kids in school, mortgage, gas, etc. Who cares.

Me: 1, Gov 0.

Training should be part of the bail-out program. Insurance should pay for customer service training. After all, they make life and death decisions for all of us. Customer Service Training should be an easy one; and the trainers should have lots of customers.

—

For more resources about training, see the Training library.

Posted on

How to start looking forward? – a dilemma

A human-like toy moving on stairs concept

James is a recently-appointed director on the board of a family business. He is the nephew of the founder and has worked in the business for several years since completing his MBA.

He is concerned because the board meetings are all taken up with historical reports and endless discussions of ‘who did what’ and how it affected performance. There has been no forward-looking or externally generated information presented to the board in all the months he has been a member.

James knows that good boards operate at a strategic level and that his industry is changing rapidly with new technology and impending legislative changes. He has attempted to raise the issue with his uncle, who is now the company chairman, but got brushed off with the statement that none of the other directors were complaining.

What should James do?

Many readers of this blog will be familiar with my newsletter The Director’s Dilemma. This newsletter features a real life case study with expert responses containing advice for the protagonist. Many readers of this blog are practicing experts and have valuable advice to offer so, for the first time, we are posting an unpublished case study and inviting YOU to respond.

If you would like to publish your advice on this topic in a global company directors’ newsletter please respond to the dilemma above with approximately 250 words of advice for Graham. Back issues of the newsletter are available at http://www.mclellan.com.au/newsletter.html where you can check out the format and quality.

The newsletters will be compiled into a book. If your advice relates to a legal jurisdiction, the readers will be sophisticated enough to extract the underlying principles and seek detailed legal advice in their own jurisdiction. The first volume of newsletters is published and available at http://www.amazon.com/Dilemmas-Practical-Studies-Company-Directors/dp/1449921965/ref=sr_1_1?ie=UTF8&qid=1321912637&sr=8-1

What would you advise?

Julie Garland-McLellan has been internationally acclaimed as a leading expert on board governance. See her website atwww.mclellan.com.au or visit her author page athttp://www.amazon.com/Julie-Garland-McLellan/e/B003A3KPUO

Posted on

What are your organization’s “Broken Windows”?

Broken glasses on the floor

Political scientist James Q. Wilson recently passed away at the age of 80. Wilson and co-author George L. Kelling argued in a landmark 1982 article in The Atlantic that communities must address minor crimes and their effects, such as broken windows, to prevent larger problems from developing. They argued the crime of vandalism wasn’t as damaging as the message the broken window sent about the community, leading to more serious crimes there.

While Wilson’s work has greatly changed policing in America, his insights have great bearing on how leaders should look at issues within their organizations. Whether the leader is looking at the purported big thing, such as an ethics issue or a strategic imperative, it is the small things that will effect the desired change. Big changes will only be implemented successfully if the community is ready to accept it. In blighted communities Wilson saw the power of fixing windows so citizens could begin to feel enough civic pride that they cared about stopping crime. In every organization there is an equivalent hot button like broken windows that needs to be addressed before leadership will have the receptive audience they seek.

—————————————————————————————–

David Gebler is the President of Skout Group, an advisory firm helping global companies use their values to clear the roadblocks to performance. David’s book, The 3 Power Values is now available. Send your thoughts and feedback to dgebler@skoutgroup.com.

Posted on

Will Machines Always Make Life Easier?

A woman making use of a POS machine

Some people were much in hate with machines then–except the owners and inventors.

There was a time when we stopped looking at machines as making our lives’ easier and started looking at the jobs they were replacing–and, indeed, they began replacing jobs. Some people were much in hate with machines then–except the owners and inventors.

My apologies to those of you who develop training apps and sell the same. Your apps have commercial purposes tied to the trainer and the company, and of course, may be very useful now. As trainers, a duty to do the best job you can to train staff and other employees in the company.

My questions are:

  1. What we are doing, and
  2. How long do we keep doing it over time.

Many writers extrapolate to see a future world. Don’t be offended but try to see the possibilities. Maybe we can stop bad news before it happens.

Today is different, however, the previous still happens. Machines do make lives’ easier and some machines do replace men or women. Lately there is a lot of commerce made from machines that can help train a man or woman better than a trainer, and in a manner in which they do it best. Some of us techies are just more comfortable with apps.

Where does it stop? When a machine can’t train a man to think? Nope, already done. Simulators can help a man practice making the correct the right decisions, in the immediacy in which it needs to be made.

Let’s back up a minute. What about games? You’re getting it. Smart Phones. Learning tools. Those learning tools that help us take control of the training. Of our training. Ultimately it tells who, what and when it is smart to talk. Now, I’m worried. When does man and woman stop controlling the machine and letting the machine control their every move. We have science fiction…or fact.

I wrote a novel about when we, Man and Woman, are so frustrated with the current world of politics that we created a machine to make those difficult decisions on our behalf. It the database was all the science, literature, history and art we would ever need.

Scientists glimpsed a doomed planet. Ineffective population controls spelled disaster for the entire world. Billions died in famines. Hundreds of millions died from air and water pollution. Still more millions died from criminal violence in their own overcrowded neighborhoods. Surprisingly global warming and rupturing the ozone layer in the earth’s atmosphere never created the disasters that scientists had been predicting for the last century. For once, politicians were willing to listen to the scientists.

Even though democratic governments were willing to give up governing themselves in order to feed the people, it was too little commitment, too late. With the world in chaos, it was up to the technically superior United AmerEuro States to take charge of the devastated planet. For the world to survive, it needed tough laws and other controls of astonishing magnitude for now there wasn’t much room for error when managing limited resources and fragile ecosystems. Efficient management was needed. More than what was humanly possible. There was simply no time for creative politics. Humans by themselves could only fail.

So much for politics and science. For those of us engaged in commerce, we made best use of machines, too, selecting perfect clients and using the perfect software to train the company.

Not much work to do now. Might as well stay home. Go on wonderful fantasy vacations; no need for real ones with the hassles of traveling and dangers of the real world.

We forgot sometime and somewhere to keep talking. Soon the computer put us in touch in our own company with the perfect person to talk to, or told us itself. We trusted it, didn’t we. We assumed it was always right. We forgot differences.

Now, any time we wanted to talk with someone socially, we had to be cleared. The computer checked our files in case we were a “danger” to the company when paired with this individual. The computer was doing the job we gave him…it. It felt like a “him.”

Then came the splintering of society… Some people thought machines shouldn’t be the ones telling us anything. They even refused the use of the “smart” machines, while others thought, “Hey it makes my life easy, what harm is there.” Or, “We can co-exist.” Or, “Take the best of both worlds.” Besides, it’s efficient for government, science, and people. We truly ruled the world. We discovered we are gods–each of us. However, the machines are smarter in every way. Maybe, we should worship them, and some do.

Wow, some heady stuff.

What happens when what is good for the machines is not necessarily good for us? What happens when a machine is worthless–worn-out? We throw it out or take the time to recycle. What about a person? No longer efficient? More efficient to train or eliminate?

Scary stuff now. I haven’t touched on artificial intelligence! A machine that thinks for itself. Where does program stop (therefore control) and self-machine control begin? So my novel begins.

—

For more resources about training, see the Training library.

There are ideas and adventure from the Cave Man, and few from the author of IN MAKR’S SHADOW or SHADOW WARRIORS, working title. Tell me which title or others you think might better get your attention. My website is still the same with more information and ideas than I can keep straight sometimes. The Cave Man Guide To Training and Development is still a people-oriented treatise on our world of training and development from the people we expect to do the work.

Happy Training.

Posted on

Why Complaints Are Gifts

Workplace colleagues discussing work

Let’s face it; it’s no fun when someone complains. Some people, the saying goes, just like to complain. Best to give them what they want and send them away, right? Not really. Complaints can provide such valuable market feedback that you’ll want to include strategies in your business plan for dealing with them. Most of the time, when customers complain, there’s a germ of truth — and useful information — in what they say.

My father-in-law, a former business executive of a large communications company, says his company viewed complaints as invaluable market information that was often more useful than focus groups, and MUCH cheaper. Customer feedback is how you build your business.

Continue reading “Why Complaints Are Gifts”