It is now more important than ever to plan how your organization will evaluate what it proposes to do.
Attend this workshop to better understand the relationships between program development and the evaluation process and to learn how to develop tools that determine the community impact of your program.

(The following was posted on behalf of Valerie Leonard.)

Welcome! This is Valerie Leonard, and I’m going to facilitate this morning’s online discussion. I believe you should have received a bio about me, but in case you didn’t, I’m a Community Development consultant with a mission to with a mission to strengthen the capacity of organizations to make a positive impact on the communities they serve through technical assistance, specialized workshops, resource and organizational development and project management. I provide technical assistance to municipalities and nonprofit organizations through the Illinois ResourceNet (IRN).

Today’s topic is The Measure of Success. We'll discuss such topics as the theory of change, needs assessment and program logic model. We’ll also discuss the relationships between program development and evaluation, evaluation planning, and how the program logic model may be used in the planning, implementation and evaluation phases of program development. The most important questions will be the ones that you ask. I'm looking forward to a very rich discussion.

Feel free to stay glued to your screen between 10 and noon, or check in periodically when you see a post that intrigues you.

To get us started today, I wanted to answer a question that came in on the message board a couple of days ago. The question was “What is a theory of change, and how does it relate to program evaluation?”

A theory of change describes a process of planned social change, including assumptions that guide its design to the long-term goals it seeks to achieve*.

A theory of change:
- Maps a process of change
- Establishes a blue print for the work ahead
- Anticipates its likely effects
- Reveals what should be evaluated - When & How

* Mapping Change: Using a Theory of Change to Guide Planning and Evaluation, published by GrantCraft, a project of the Ford Foundation.

Kelsey Holsapple wrote:
It is now more important than ever to plan how your organization will evaluate what it proposes to do.
Attend this workshop to better understand the relationships between program development and the evaluation process and to learn how to develop tools that determine the community impact of your program.

Kelsey,
I'm looking forward to the discussion tomorrow!

Working through some logistical items with our facilitator. I'll post this but delete later...

Good morning, everyone. By now you should have had the opportunity to review the e-mail message and our "opening statements" (smile)regarding the topics to be discussed. I thought we'd jump into our live discussion. Let me know if you have any burning questions regarding evaluations, program logic models, how they are used in program planning, implementation and evaluation, etc. You may even have other questions that I haven't listed.

I have found in my experience that evaluation is one of those topics that people have a difficult time "wrapping their arms around." People often feel that they don't know enough about the topic to even ask a question. The fact that no two funders frame questions regarding evaluation the same way further complicates things.

At the risk of sounding patronizing, have you had similar experiences? What has been your experience with responding to evaluation questions, or preparing for an evaluation?

Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

Great start, Rosemary. Can you be more specific? Are you starting a new program, or you have an existing program?

Valerie F. Leonard wrote:
I have found in my experience that evaluation is one of those topics that people have a difficult time "wrapping their arms around." People often feel that they don't know enough about the topic to even ask a question. The fact that no two funders frame questions regarding evaluation the same way further complicates things.

At the risk of sounding patronizing, have you had similar experiences? What has been your experience with responding to evaluation questions, or preparing for an evaluation?

A question: Is there a way that we can prepare somehow for evaluations over a series of proposals. I mean, we've discussed the concept of a 'cheesebook,' collecting the pertinent information that is likely to be requested as part of all proposals. Can evaluations be addressed in the same way?

Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

I'll defer to Valerie first but not sure what the references to 'board code and smiley buttons' mean.

Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

Okay, now I have some specific thoughts about your initial query. My experience, and I suspect yours, shows that when you consistently use a logic model when starting a new venture, the relationship between program development and evaluation will be obvious because you're effectively forced to deal with that relationship at the outset. I mean, using a logic model asks you to include all inputs and desired outcomes, so in respect to the latter you need to determine how you'll know if your outcomes are being achieved - thus evaluation.

And when I say 'outcomes' I refer to much more than a count of people in the room. Funders, public and private, increasingly require demonstrated impact which the number of participants does not convey.

Deborah Callaghan wrote:
Kelsey Holsapple wrote:
It is now more important than ever to plan how your organization will evaluate what it proposes to do.
Attend this workshop to better understand the relationships between program development and the evaluation process and to learn how to develop tools that determine the community impact of your program.

Kelsey,
I'm looking forward to the discussion tomorrow!

Deborah,
Putting you on the spot... what especially are you hoping to elicit from the discussion that prompts your enthusiasm?

Valerie F. Leonard wrote:
Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

Great start, Rosemary. Can you be more specific? Are you starting a new program, or you have an existing program?

I haven't heard back, but I will take a crack at answering your question.

Regardless of whether you have a program in place, or starting a new program, a good place to start is a theory of change. As I mentioned in the opening, A theory of change describes a process of planned social change, including assumptions that guide its design to the long-term goals it seeks to achieve
-Map of a process of change
-Establishes a blue print for the work ahead
-Anticipates its likely effects
-Reveals what should be evaluated
-When
-How

I have attached a sample theory of change map. I will provide a worksheet in a separtate post.

I thought folks might be interested in the visual image of the theory of change we have used as the basis for goodWORKSconnect. Our vision is in the middle with the strategies attached. What we are continuing to work on is the measures for how we know if we are successful.

Earl Bricker wrote:
Deborah Callaghan wrote:
Kelsey Holsapple wrote:
It is now more important than ever to plan how your organization will evaluate what it proposes to do.
Attend this workshop to better understand the relationships between program development and the evaluation process and to learn how to develop tools that determine the community impact of your program.

Kelsey,
I'm looking forward to the discussion tomorrow!

Putting you on the spot... what especially are you hoping to elicit from the discussion that prompts your enthusiasm?

I need input regarding evaluations. I've noticed that people measure "success" by very different criteria depending upon their personal biases and needs. However, when working with funders, objective measures need to be used so that they know, and we know, that we're moving in a positive direction.
I am involved in sustaining two historical museums that display artifacts donated by local community members. Some in the community think that we should devoted more time to tourism development, while others feel that serving the needs of the local community and those in the surrounding areas should be the primary focus. I think both of these need to be targeted.
Which of these groups do you think most funders would emphasize? I'm not suggesting that we alter our mission to please funders. Our mission is to collect, preserve, and display artifacts pertaining to our local heritage. Can't we do this, and focus on attracting both groups?

Earl Bricker wrote:
Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

Okay, now I have some specific thoughts about your initial query. My experience, and I suspect yours, shows that when you consistently use a logic model when starting a new venture, the relationship between program development and evaluation will be obvious because you're effectively forced to deal with that relationship at the outset. I mean, using a logic model asks you to include all inputs and desired outcomes, so in respect to the latter you need to determine how you'll know if your outcomes are being achieved - thus evaluation.

And when I say 'outcomes' I refer to much more than a count of people in the room. Funders, public and private, increasingly require demonstrated impact which the number of participants does not convey.

You are absolutely right about outcomes being more than the number of people in the room.

If you are looking at a process evaluation, in which you are evaluating efficiency of a program, production (ie, the number of people served) could be a very meaningful number. However, when you are talking about outcome evaluations, it is more important to speak of the impact you will have on the clients or community.

For example, what is the environment like? What's going on at a macro level? ie, government. What's going on in the community? How will these changes impact our ability to deliver services?
Given the environment? What are our overall goals? What tools will we need? (ie, staff, ) What activities will we engage in in order to achieve the goals? What will change?
Will people have more knowledge or changed behaviors?

I have posted an example from the Second Chance Act.

Deborah Callaghan wrote:
Earl Bricker wrote:
Deborah Callaghan wrote:
Kelsey Holsapple wrote:
It is now more important than ever to plan how your organization will evaluate what it proposes to do.
Attend this workshop to better understand the relationships between program development and the evaluation process and to learn how to develop tools that determine the community impact of your program.

Kelsey,
I'm looking forward to the discussion tomorrow!

Putting you on the spot... what especially are you hoping to elicit from the discussion that prompts your enthusiasm?

I need input regarding evaluations. I've noticed that people measure "success" by very different criteria depending upon their personal biases and needs. However, when working with funders, objective measures need to be used so that they know, and we know, that we're moving in a positive direction.
I am involved in sustaining two historical museums that display artifacts donated by local community members. Some in the community think that we should devoted more time to tourism development, while others feel that serving the needs of the local community and those in the surrounding areas should be the primary focus. I think both of these need to be targeted.
Which of these groups do you think most funders would emphasize? I'm not suggesting that we alter our mission to please funders. Our mission is to collect, preserve, and display artifacts pertaining to our local heritage. Can't we do this, and focus on attracting both groups?

You can segment the market, however, you should consider doing a needs assessment beforehand. A needs assessment would help you to quantify the demand in each sector. You can conduct an environmental scan to get a sense for who your competitors and partners are. You can conduct surveys of potential users to get a sense for what they might want to see. Obviously the local community may have different tastes than people in other areas.

The group the funder emphasizes depends upon the funder's own philosophy, goals and objectives. Funders give money to carry out the goals and objectives of their programs. The more closely aligned your goals and objectives are to the funders, and the better you can document outcomes, such as changes in knowledge, or whatever the funder would like to see changed the better. It is important to avoid mission creep.

Valerie -
In a previous reply you mentioned the need to avoid mission creep. How would you define that? I think I understand because it seems intuitive but would like your definition, if you would.

Annie Hernandez wrote:
I thought folks might be interested in the visual image of the theory of change we have used as the basis for goodWORKSconnect. Our vision is in the middle with the strategies attached. What we are continuing to work on is the measures for how we know if we are successful.

File Attachment:
File Name: theory_change-20100728.pdf
File Size: 158132

In looking at the theory of change that you use, the term "planned social change" strikes me. In the last three months, more people from the local community and out of state/out of town have visited our museums with specific requests. The number of visitors to one museum
is up by about 150 from this time last year. These are measures of success for me. I've written weekly articles in the local newspaper to facilitate more historical discussions among the local community members. Judging from comments made to myself and others, this seems to be working. Social change is occurring!
Tourists are interested in the museums, but those primarily invested in them are the local community members. People who
have moved and are back to visit are very invested.
I've been advised to advertise our museums in newspapers as far away as 75 miles or more. I have also received recommendations to focus on closer, surrounding areas.
The objective is to attract people to the museums, and stimulate an interest in history, and a greater sense of unity and involvement within the community. However, the
ways in which to do this vary. There are those in the community who are very uncomfortable with the high numbers of tourists at times; they would like tourism to decrease. However, tourism is necessary for the economic development of the community. The leaders in the historical society recognize the need to address all of these needs; those in the community don't always agree.

Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

When we develop programs, we are usually responding to an identified need in the community. In an ideal setting, we will have completed a theory of change, which will cause us to review our mission, and assumptions about the environment in which we operate. We will develop goals and objectives, identify our targets and strategies for reaching the goals and objectives as well as define outcomes so we know that we can tell if we are successful.

The information from the theory of change will inform the needs assessment. When we do the needs assessment, we determine the lay of the land, determining who is meeting the needs we identified, and assessing the extent to which the needs are being met, or any gaps in services.

This information may be used to develop program logic models. Program logic models, as I said before, help you describe the environment, develop goals and objectives, identify the resources and activities, and desired outcomes. The indicators will help us to measure our progress to the goals. More detailed logic models will identify research methods and data sources. These data are used to develop more detailed programs.

The link between the planning, implementation and evaluation process is found in the program logic model. The program logic model, as a planning tool, you are using the model to literally test the model's logic. You consider the alignment of goals, objectives, and outcomes, giving them a reality test. You begin to identify measures and data sources from the start, so that program management and evaluation are seamless.

During the implementation phase, you use the logic model as a tracking tool. Management focuses on the relationships between the activities and the desired outcomes.

During the evaluation phase, you use the logic model to lay the groundwork for the formal evaluation. Ideally, the stakeholders have all decided how to use the information and report the results.

Here are resources you may find to be helpful.

Here is an awesome resource from the Kellogg Foundation. If you follow the steps in this book, your organization either will not need to hire an outside evaluator, or you will save lots of time and money, because you will be in a better position to manage the process.

Deborah Callaghan wrote:
Annie Hernandez wrote:
I thought folks might be interested in the visual image of the theory of change we have used as the basis for goodWORKSconnect. Our vision is in the middle with the strategies attached. What we are continuing to work on is the measures for how we know if we are successful.

File Attachment:
File Name: theory_change-20100728.pdf
File Size: 158132

In looking at the theory of change that you use, the term "planned social change" strikes me. In the last three months, more people from the local community and out of state/out of town have visited our museums with specific requests. The number of visitors to one museum
is up by about 150 from this time last year. These are measures of success for me. I've written weekly articles in the local newspaper to facilitate more historical discussions among the local community members. Judging from comments made to myself and others, this seems to be working. Social change is occurring!
Tourists are interested in the museums, but those primarily invested in them are the local community members. People who
have moved and are back to visit are very invested.
I've been advised to advertise our museums in newspapers as far away as 75 miles or more. I have also received recommendations to focus on closer, surrounding areas.
The objective is to attract people to the museums, and stimulate an interest in history, and a greater sense of unity and involvement within the community. However, the
ways in which to do this vary. There are those in the community who are very uncomfortable with the high numbers of tourists at times; they would like tourism to decrease. However, tourism is necessary for the economic development of the community. The leaders in the historical society recognize the need to address all of these needs; those in the community don't always agree.

Deborah,
Thanks for the note! Good news for your increase in traffic. Can you point to one thing you think is really working to help with this?

One thing that may be helpful is having your board aligned with your vision and strategies you are implementing to get there...so everyone is on the same page.

Something else that may be of interest is to get together or chat with the tourism people in your region...they usually have a good grasp on what works for their areas. Another thing tourism is good at is quantifying the economic impact of visitors (especially from out of town) to attractions...money spent in the area adds up fast! Here's a link to the Illinois Tourism if you haven't already reached out:
www.enjoyillinois.com/home.aspx

Valerie F. Leonard wrote:
Deborah Callaghan wrote:
Earl Bricker wrote:
Deborah Callaghan wrote:
Kelsey Holsapple wrote:
It is now more important than ever to plan how your organization will evaluate what it proposes to do.
Attend this workshop to better understand the relationships between program development and the evaluation process and to learn how to develop tools that determine the community impact of your program.

Kelsey,
I'm looking forward to the discussion tomorrow!

Putting you on the spot... what especially are you hoping to elicit from the discussion that prompts your enthusiasm?

I need input regarding evaluations. I've noticed that people measure "success" by very different criteria depending upon their personal biases and needs. However, when working with funders, objective measures need to be used so that they know, and we know, that we're moving in a positive direction.
I am involved in sustaining two historical museums that display artifacts donated by local community members. Some in the community think that we should devoted more time to tourism development, while others feel that serving the needs of the local community and those in the surrounding areas should be the primary focus. I think both of these need to be targeted.
Which of these groups do you think most funders would emphasize? I'm not suggesting that we alter our mission to please funders. Our mission is to collect, preserve, and display artifacts pertaining to our local heritage. Can't we do this, and focus on attracting both groups?

You can segment the market, however, you should consider doing a needs assessment beforehand. A needs assessment would help you to quantify the demand in each sector. You can conduct an environmental scan to get a sense for who your competitors and partners are. You can conduct surveys of potential users to get a sense for what they might want to see. Obviously the local community may have different tastes than people in other areas.

The group the funder emphasizes depends upon the funder's own philosophy, goals and objectives. Funders give money to carry out the goals and objectives of their programs. The more closely aligned your goals and objectives are to the funders, and the better you can document outcomes, such as changes in knowledge, or whatever the funder would like to see changed the better. It is important to avoid mission creep.

I've worked on surveys-they are useful! People identify different needs depending on their perspectives. Need to address these without losing sight of our mission.

Here is a great evaluation plan template used by one of the federal government agencies. The only sections I would add would be sections on budget and timeline.

Annie Hernandez wrote:
Deborah Callaghan wrote:
Annie Hernandez wrote:
I thought folks might be interested in the visual image of the theory of change we have used as the basis for goodWORKSconnect. Our vision is in the middle with the strategies attached. What we are continuing to work on is the measures for how we know if we are successful.

File Attachment:
File Name: theory_change-20100728.pdf
File Size: 158132

In looking at the theory of change that you use, the term "planned social change" strikes me. In the last three months, more people from the local community and out of state/out of town have visited our museums with specific requests. The number of visitors to one museum
is up by about 150 from this time last year. These are measures of success for me. I've written weekly articles in the local newspaper to facilitate more historical discussions among the local community members. Judging from comments made to myself and others, this seems to be working. Social change is occurring!
Tourists are interested in the museums, but those primarily invested in them are the local community members. People who
have moved and are back to visit are very invested.
I've been advised to advertise our museums in newspapers as far away as 75 miles or more. I have also received recommendations to focus on closer, surrounding areas.
The objective is to attract people to the museums, and stimulate an interest in history, and a greater sense of unity and involvement within the community. However, the
ways in which to do this vary. There are those in the community who are very uncomfortable with the high numbers of tourists at times; they would like tourism to decrease. However, tourism is necessary for the economic development of the community. The leaders in the historical society recognize the need to address all of these needs; those in the community don't always agree.

Deborah,
Thanks for the note! Good news for your increase in traffic. Can you point to one thing you think is really working to help with this?

One thing that may be helpful is having your board aligned with your vision and strategies you are implementing to get there...so everyone is on the same page.

Something else that may be of interest is to get together or chat with the tourism people in your region...they usually have a good grasp on what works for their areas. Another thing tourism is good at is quantifying the economic impact of visitors (especially from out of town) to attractions...money spent in the area adds up fast! Here's a link to the Illinois Tourism if you haven't already reached out:
www.enjoyillinois.com/home.aspx

Thanks for the encouragement, Annie! I think the articles in the paper are helping, my presence in the community is helping ( I talk with different people in the community regularly, and our listing in a travel brochure is helping. I think the board is in agreement that we need to focus on the local community while attracting the tourists at the same time.
I am working with the city tourism office. Currently, tourism is up due to a very specific event in town. When this ends, the numbers will be down. The city has need for increased overnight stays; our museums seem to be attracting those who are staying for a few days and those who are taking day trips.

Rosemary Gibbons wrote:
Can we discuss the relationships between program development and evaluation, evaluation planning?

I need evaluation techniques.
Please note: although no board code and smiley buttons are shown, they are still usable.

When you say evaluation techniques, are you speaking of data collection methods, or types of evaluation? If you are speaking of research methods/data collection, you can use surveys, assessments, interviews, literature reviews, asset mapping, etc.

The two most common evaluation techniques are process evalation and outcome evaluation. Sometimes the two types are combined.

Process Evaluation (Formative Evaluation) is concerned with how the program is delivered.
-Focuses on nuts and bolts program operations
Ie, when the program activities occur, where they occur, and who delivers them. (Think of a program flow chart)

-Asks the question “Is the program being delivered as intended?”

-May be used to measure operational efficiencies and recommend
more streamlined processes, cost savings
-Example: Removing duplication in the organization or
program structure

-Outcome Evaluation (Summative Evaluation) addresses the question of what are the results/changes over time.
Commonly stated in terms of short-term outcomes and long-term outcomes.

-For example, in an exercise program, a short-term outcome could be a change knowledge about the health effects of exercise, or it could be a change in exercise behavior. A long-term outcome could be less likelihood of dying from heart disease.

-Often used to measure programmatic impact upon clients and community; make recommendations for improvements; determine feasibility of continuing programs.

You might also consider a strategic plan to help you position yourselves for change over the long term. An evaluation in an of itself is limited to help you ascertain the business and markets you should be in. In an ideal world, you need to do a theory of change which will inform a needs assessment, strategic plan, program logic model, program design and evaluation. One process builds upon another.

Valerie is composing a final summary message, and in the meantime I wanted to ask if there are any final questions or comments to address in these last few minutes.

This is the final outing in the six-session series. This and the others are all resident in the Forum so they can be re-visited to read the messages and download any resources shared by the various facilitators and participants.

We will be soliciting feedback from all who signed up, whether they participated during the actual two-hour session, to determine if this method of providing technical assistance and information is useful and something to continue. And if so, how to improve the format so that people gain even more value.

Thanks for your participation today. I've enjoyed being part of this initiative and look forward to future such endeavors.

And a special thanks to the Lumpkin Foundation and goodworksconnect.org for partnering in this experiment and providing the infrastructure to allow University of Illinois Extension and Illinois ResourceNet to reach audiences in this new way.

Thank you for your participation. Your comments underscore the fact that evaluation is not an end unto itself. Nor may it successfully be done in a vacuum. Evaluation is the culmination of a number of organizational development processes, including development of a theory of change, needs assessment, development of a strategic plan, development of a program logic model, development of a program, which will in turn, be implemented and evaluated. The process is very similar to learning math. Each process or concept builds upon processes and concepts that were completed before. Unfortunately, very few organizations have the time, expertise or funding to do this comprehensively.

In closing, I hope the materials I've shared will be useful as you learn more about the evaluation process. Good luck,and good bye.

Valerie F. Leonard wrote:
You might also consider a strategic plan to help you position yourselves for change over the long term. An evaluation in an of itself is limited to help you ascertain the business and markets you should be in. In an ideal world, you need to do a theory of change which will inform a needs assessment, strategic plan, program logic model, program design and evaluation. One process builds upon another.

I'll review this information. I can see how one process would inform and guide the direction of the next ones. Can't your needs assessment guide the theory of change, too? This seems overwhelming right now, but I can see that once these pieces are in place, evaluation procedures could be ongoing and very illustrative.