Blog

Logic Models: A New Opportunity

Hey everyone, I’m Phil Stoeklen! I’m excited to share some thoughts on a tool we leverage a lot with our stakeholders: logic models. Logic models are useful tools that help program stakeholders and evaluators gain an understanding of the strategic model or vision for their initiative. Logic models can help to map out what resources are needed to sustain program activities, and can also assist with visualizing  the outcomes and impacts of program activities…something that can be difficult to describe in mere words (especially when you have a lot of stakeholders to consider). The question I would like to pose, however, is do they actually help understand all outcomes and impacts?

If you think about it, the way that we organize logic models primes us to miss unintended (occasionally negative) program effects. Programs are, after all, treatments for existing problems. Like any treatment, there is always a potential risk for negative side-effects. By ignoring these potential adverse impacts when we lay out program logic, we open ourselves up to not catching problems as early as we can–or in the worst-case scenario, after it is simply too late. 

I think this is a commonly encountered phenomenon in logic model design, and I think it is because of a couple of important reasons. First, when programs are conceptualized the goals and imagined impacts are often quite lofty (not necessarily a bad thing), and almost always positive (again, not necessarily a bad thing). It would be quite odd, afterall, to design a deliberately harmful program, but it is worth pointing out that the negative considerations are not often a focal point of logic modelling sessions. A second reason for this is really a combination of the first observation and the literal format of logic models. We talk about inputs, outputs, outcomes, and impacts, but the latter two are almost never facilitated from the perspective of how this program design could conceivably harm populations. 

Now, having ambitions for programs to have wonderful long-term impacts is great! It helps program stakeholders  set big goals for themselves and fosters evaluation touch-points by identifying potential areas to measure for effect. The bigger problem that is happening is really a consequence of not thoroughly exploring what is realistic to expect as an outcome and impact of said program, and conversations about how we will react if/when something goes awry.

With this observation in mind, don’t you think it is time we have a real conversation about how we model program logic, and how we help our clients understand and anticipate as many program effects as we can?  It isn’t about focusing on the negative…instead, it’s having an informed conversation about how we recognize that every treatment has the potential for both positive and negative effects. In future posts, we will share some strategies to incorporating this important (but often missed) element of logic models. 

Phil is a Senior Managing Consultant at Viable Insights, where he leverages his strong background in evaluation. He has a Master’s degree in Applied Psychology with concentrations in Health Promotion and Disease Prevention, Evaluation Research, and Industrial-Organizational Psychology. Phil has been an evaluator and project manager on multiple projects, including: comprehensive needs assessments, community perception projects, formative and summative program evaluations, and impact evaluations. His projects have ranged from short-term to multi-year, and has collectively worked on more than $23 million in both grant and privately funded programs/initiatives. Clients he has worked with include Margaret A. Cargill Philanthropies, U.S. Department of Labor, University of Wisconsin System, the Wisconsin Technical College System, and the Annie E. Casey Foundation, among others. In addition to Phil’s professional consulting experience, he serves as an instructor in the Evaluation Studies and Institutional Research graduate certificate program at the University of Wisconsin- Stout. In that capacity, he teaches courses covering evaluation theory, data collection techniques and best practices, and evaluation applications. Whether in his role as an evaluator or instructor, his goal remains the same —  providing individuals and organizations with the tools, skills, and capacity to collect and use data in their decision making process. Find him on LinkedIn or Twitter!

Process Mapping: The First Meeting

Hello! Levi Roth, here – back to share more on processing mapping! If you’ve had a chance to read our previous post, A Quick Introduction to Process Mapping, this new post will serve as a continuation of our discussion. If you haven’t had an opportunity to read this quick introduction to process mapping, I encourage you to check it out before digging into this week’s blog!

Now that you’ve had a chance to be introduced to the general idea of what process mapping is, I want to share some of my experiences with conducting the initial meeting. Here are five key things I’ve learned:

  • Make sure you have the key stakeholders in the room when going through the process.
    • You want to make sure that everyone in the room can accurately and enthusiastically speak to their role within the process.
  • Make sure you take some time to introduce your team members to process mapping.
    • It’s important everyone is comfortable! There’s nothing worse than trying to hold a process mapping meeting and looking out at a sea of blank stares.
  • Keep the team on task for your meeting.
    • One thing I’ve noticed during the As-Is process mapping meeting is that some team members like to try and jump ahead to the To-Be process. It’s crucial to keep them focused on the current actual process to flush out issues.
  • Allow for an appropriate amount of time to conduct your meetings.
    • It is very unlikely that you’ll be able to accurately document your process in 30 minutes or even an hour (unless you’re dealing with process mapping rock stars!). I typically schedule my meetings for two hours and let the team know ahead of time that we may not wind up using the full two hours. But from my experience, we almost always need the two hours (if not more).
  • Do not allow too much time to pass between your As-Is process mapping meeting and To-Be process mapping meeting.
    • Ideally, I like to schedule the To-Be process mapping meeting within one week of completing the As-Is process map. The idea behind this is that you don’t want the stakeholders to lose interest or their valuable ideas on how to improve the process.

Key Stakeholders

I tend to work closely with the process owner to make sure we have all of the necessary individuals within our meetings. Remember, the process owner should be someone who is a Subject Matter Expert (SME), can be considered a champion of the process, and has the ability to effectively communicate with other roles within the process. The main reason you want to make sure you have everyone in the room that can speak to their roles and responsibilities is that you can accurately and effectively identify the current process. Believe me…they will be more than willing to share some of their pain points within the process or describe some of the gaps or inefficiencies. What you do not want to happen is have someone who is speaking to a role or responsibility that they are not currently in.

Make Time for Process Mapping Overview

This is something I learned early on when I began my adventure with process mapping. During some of my early process mapping meetings I went in with the assumption that my key stakeholders had knowledge and experience around this. This mistake was definitely a lesson learned for me. Shortly after these missed opportunities, I developed a quick introduction to process mapping presentation that I go through during every initial meeting. Being able to establish what process mapping is, the goals for the meeting, and walking the team through a simple process map example (buying groceries is my go-to example) helps put everyone in the right frame of mind before working on their process. Ever since I created that intro presentation, my initial process mapping meetings have gone much smoother. I’ve also received feedback from team members that they really enjoyed and appreciated the introduction because they had no idea what process mapping was.

Keeping the Team on Task

This is especially important when you are trying to document the actual As-Is process map. Once the team really starts to get in the groove, they might begin to introduce new ideas to improve the process. Don’t get me wrong, this is great that they are starting to see ways to create improvement in their process! However, it’s best to save that idea in a parking lot and circle back around to it when the team begins to create a To-Be process map. I’ve also experienced some meetings where team members will provide tasks in the process and then follow it up with, well that’s what is supposed to happen anyway. This is a great time to ask a follow up question to understand what is actually happening in the process instead of what is supposed to be happening.

Proper Amount of Time for Meetings

Process Mapping takes a good amount of time to complete. Especially if you want to start the first meeting with an introduction to process mapping and establishing goals for the meeting. I typically will try and schedule a two-hour meeting for the As-Is and To-Be process mapping meetings. I’ll also inform the team that we may not need the full two hours and sometimes I’ll need to schedule an additional meeting to finalize the maps. This can be dependent on how complex the process is and how quickly and accurately the team can describe and go through their process. I’ve found that it’s better to schedule more time than you need in order to avoid running out of time when the team is in the right mindset.

Proper Amount of Time In-between Meetings

The final key point I want to mention is making sure you do not allow too much time to pass between your As-Is and To-Be process mapping meetings. You want to make sure your team members are able to stay in the mapping mindset. My current best practice is to schedule these meetings a week apart, at the most. I’ve found that if we aren’t able to reconvene until a week or more after the initial meeting, we spend more time trying to get into the correct state of mind. Additionally, some of the great ideas that team members had on improving the process were lost during the time in-between meetings. I understand that it can be difficult planning these meetings less than a week apart but the payoff is worth it!

These are five key things I have learned when scheduling and conducting my process mapping meetings. As you reflect on process mapping, what have you found to be essential to an efficient and effective process mapping meeting?

Levi graduated with a Master’s Degree in Applied Psychology with concentrations in Industrial – Organizational Psychology and Evaluation Research. Before completing his Master’s Degree, he was hired as a Project Manager on a $23 Million grant evaluation project within Wisconsin. During this time, he gained a wealth of experience in: Project Management, Resource Management, Evaluation, Analytics, Data Visualization, Process & Program Improvement, and many other skills. After the completion of the grant evaluation, Levi began a new adventure as an Agency Wide Business Project Manager for a large Government Agency within Minnesota (and he never heard the end of it from his Wisconsin friends ). He has been in his current role for roughly two years and loves the new experiences and challenges he is faced with on a daily basis. Looking to nerd out about process improvement or government work? Levi is your guy!

Interested in having us facilitate a process mapping session for your organization? Let’s chat.

Stay tuned for future posts on process mapping, including tools you can use for facilitating meetings and designing maps!

A [Kind of] Quick Introduction to Process Mapping

If you’ve ever caught yourself saying any of the following things:

  • There has to be a better way of doing this.
  • Ever since (insert name here) left the company we don’t know our own process anymore.
  • Wait, I’m supposed to be doing what again?

…then process mapping might be for you!

Hi, Levi Roth, here! Let me start by saying that I was drawn to process mapping as someone who has a passion for identifying and improving upon areas of opportunity within my organization. I’m excited to share insights and information I have learned along the way with those who share a similar passion or interest (or really anyone who needs a way to better understand their organization’s currents for the sake of capitalizing on strengths and addressing problematic areas. Who you are matters less, right? Because whether you’re an evaluator (internal or external), analyst, or program director, processes exist everywhere…and ultimately, we all seek to be as efficient and effective as possible. Okay, enough back story — let’s get started!

First off, let’s take a quick look at what I mean by a process map and discuss its purposes. I like to think of a process map as a planning and management tool that visually describes the flow of work. You can apply process maps to nearly anything; picking up groceries, cooking, getting ready for school/work, complex business processes, etc. If there are tasks to complete that have consecutive or sequential steps, you can map it. Creating a process map can help you and your organization in a variety of ways.

Process maps can:

  • Identify
    • Bottlenecks (i.e. a part of a process or procedure where progress is hindered within an organization)
    • Repetitions (i.e. an inefficient use of resources)
    • Delays (this one speaks for itself, right?)
  • Define
    • Process Ownership
      • Who will be ensuring that this process is properly followed? Typically these individuals are the Subject Matter Experts (SMEs) of the process, can be considered a champion of the process, and have the ability to effectively communicate with other roles within the process to make sure it is being followed efficiently.  Tasks
      • These are the specific sequential tasks that are established within the process itself. Let’s look at some of the basic tasks involved in one of the processes I’ve mentioned previously: picking up groceries! Some example tasks could be: taking inventory of groceries at home, finding recipes you want to make for the week, create a grocery list of items still needed, etc. Roles and Responsibilities
      • This part is the most crucial for the success of creating process improvement and often times is overlooked. It is important to establish clear roles within the process and the responsibilities that are associated with them.
  • Provide Insights
    • Having the key stakeholders in the room at the same time to go through the process can also help raise other issues that may be happening. This is where we can start to gather insights on potential issues within the process.
  • Allows Brainstorming Opportunities for Process Improvement
    • This often times rolls into the Provide Insights portion quite well. From my experience, during the As-Is process mapping meeting individuals will already start brainstorming ideas on how to improve the process.
  • Provides Process Documentation
    • This can be incredibly valuable, especially if your organization has a high turn over rate. Once the process map is finalized, you can distribute this to others within your organization who use the process. You can also provide this documentation to new hires in order to help them better understand the process along with roles and responsibilities associated.

When documenting your processes, it is important to start with your As-Is process or What’s Happening Now. The first map you create really focuses on how the process works currently. In my experience, there are almost always three versions to any process:

What you think it is:

What the actual current state is:

What it should be:

Your organization may already have some documented process or procedures, however, as you begin to walk through what the actual current state is you might see some serious variation. By documenting how the process is actually being conducted it allows us to identify the hidden issues which may be causing those bottlenecks, repetitions, or delays we discussed before. Exposing these hidden issues allows the team to critically think about how to improve their process. This leads us to the second process map you’ll want to create with the team, and that is your To-Be process map. Because of the awareness we now have of those hidden issues, we can draw a more ideal map that streamlines tasks, clearly defines roles, and eliminates redundancies. If you are new to the concept of process mapping, I hope this brief blog has encouraged you to think about the processes in your current job and life, and the ways in which you can work to improve those process experiences. If you are a veteran process mapper, then I invite you to please reach out so we can keep the conversation going and share best practices with our community!

Levi graduated with a Master’s Degree in Applied Psychology with concentrations in Industrial – Organizational Psychology and Evaluation Research. Before completing his Master’s Degree, he was hired as a Project Manager on a $23 Million grant evaluation project within Wisconsin. During this time, he gained a wealth of experience in: Project Management, Resource Management, Evaluation, Analytics, Data Visualization, Process & Program Improvement, and many other skills. After the completion of the grant evaluation, Levi began a new adventure as an Agency Wide Business Project Manager for a large Government Agency within Minnesota (and he never heard the end of it from his Wisconsin friends 😊). He has been in his current role for roughly two years and loves the new experiences and challenges he is faced with on a daily basis. Looking to nerd out about process improvement or government work? Levi is your guy!

Interested in having us facilitate a process mapping session for your organization? Let’s chat.

Stay tuned for future posts on process mapping, including:

  • Holding the Meeting
  • Tools for Process Mapping
  • Implementing the Change

#AZENET20: Let’s Talk About Soft Skills

I’m very excited to announce that I was recently elected as the 2020-2021 President of the Arizona Evaluation Network. As someone who has been involved in the American Evaluation Association since the beginning of graduate school, finding a local affiliate was one way to stay connected to our field throughout the year. The opportunity to propel the Arizona Evaluation Network forward with a fantastic board and membership, striving for an engaging, equitable, and relevant community, is definitely an honor.

In future blogs, I will focus more on my adoration of the field of evaluation, but here I want to share the theme for the Arizona Evaluation Network’s next conference (whoo, 2020!): Soft Skills for Evaluators (and institutional planners, analysts, project managers, and the like). In our most recent Arizona Evaluation Network conference, we took a moment to reflect and reengage in the fundamentals of evaluation. This prompted me to think about what else is relevant to practitioners and academics alike – regardless of the context in which we find ourselves in the field of evaluation (or the approach we take, for that matter). Combining that with the important critical role that interpersonal skills, reflective practice, and self-awareness play in our success (or so I think), soft skills was almost an obvious choice. So, let’s talk more about that…

As a starting place, I’ll share that I’ve had this realization, which is that by bringing more of me to my practice, reducing the disparity between who I am personally and the work I do, everyone benefits (probably one of those things someone much wiser told me 1,000 times but I had to learn the hard way). So, you could say my development of my own self-awareness (a really important soft skill if you ask me!) is something that has really shaped how I engage in projects and collaborate with clients.

The next reason is that when I take a moment to consider soft skills of evaluators, I see it as a multi-faceted concept that has the potential to enrich our work and, even more importantly, play a role in the impact of the organizations we partner with. Soft skills include having the ability to navigate difficult conversations…the ones where humility and vulnerability are at the forefront. It’s the point in which we remove ourselves from a pedestal, the expert role, and come alongside stakeholders to achieve a common, people-centered goal. Soft skills enable us to get the right voices involved, which often means more active listening and less prescriptive consulting. It’s when we are focused on who we are as people, knowing where our value-add is to an organization or community, and capitalizing on the skills of others (stakeholders and other evaluators) that we facilitate the greatest impact: we are stronger together! Soft skills also prompt us to consider our approach and recognize that sometimes we need to pivot. And almost more importantly, acknowledging that it’s okay to do so, as we’re trying to tear down the wall between consultants/experts and stakeholders.

As I consider our efforts to establish the technical skills needed to be effective practitioners, I propose that we should be simultaneously focused on the other part of the formula: soft skills. Because after all is said and done, if we can’t have meaningful dialogue with stakeholders…meeting them where they’re at…it seems aspirational to think that a reliance on our technical skills alone will result in the use of findings. Let’s start thinking beyond certifications and traditional forms of expertise. You might call it back to the basics on effective human interaction!

I’m excited to share what I’ve learned about the importance of soft skills, but even more pumped to hear about what others have found to be effective in their work. Look for more details in the coming months on the 2020 Annual Arizona Evaluation Network Conference (even better, get on our mailing list to ensure you receive updates like a cool kid!).

Reflections from #AZENet19

A few weeks after the 2019 Annual Arizona Evaluation Network Conference, I felt inclined to reflect on my experience. This year’s theme, Refocusing on the Fundamentals, served as a call to get back to the roots of evaluation practice. This year’s conference really reminded me that we should never become too complacent with our skills in even the most routine of tasks. The reality is…things change – especially with environmental (i.e. situational or contextual) factors and stakeholder dynamics. We as evaluators need to flex to this for the sake of both our own development and that of the programs, collaboratives, or communities we are working with.

So as expected, I came away from the conference challenged and recharged (I love getting together with my people!); I was ready to take on a whole new set of goals! I realized that the conference theme leveraged two important aspects of how I strive to approach projects. The first is to maintain a heightened level of self-awareness, and the second (and really how I continually push for the first) is the application of ongoing reflective practice — asking myself questions like…

What are my strengths? What are my areas of opportunity? What do I enjoy doing? Where can I continue to develop? Where can I leverage others to add more value and impact to my clients’ projects?

Through my interactions with AZENet colleagues, I realized I was answering some of these questions naturally through peer discussions and a reminder of the foundational principles in our work. This experience reinforced the idea that we need to be coming together as diverse groups to enhance our practices and what we deliver. It’s how we move our field forward. Personally, I think the experience helped reaffirm how important it is to promote self-awareness and reflective practice in my work, and it also helped increase my awareness of the fact that I want/need more opportunities for collaboration my peers. Growth is difficult, so why not go through the process with other people that might be asking themselves the same reflective questions as you are.

I’m considering this my challenge to get in that space more…and want to encourage others to do the same!

Look for my future post, which will include my vision as the 2019 AZENet President-Elect.

Overlapping Column Charts: A Quick Actual v. Goal Comparison

Hello there! I’m writing you as a follow up to a workshop I recently facilitated with Nicole Huggett, MSW, for the Arizona Evaluation Network in Phoenix. A big focus of our time together was spent on covering visualization options for comparing goals and pre-post results.

One of the popular charts we discussed were overlapping column charts and how they can be used to compare actual performance to goals. Since the workshop, I have found overlapping column charts to be very valuable data visualizations for this – so much so that I knew I had to share the steps publicly (OKAY, I also kept getting asked for the steps, so I knew writing it once would save us all some time!).

Although I already shared when you might use this chart, the particular scenario I was to set is related to survey participation. Specifically, one community organization needed a quick way to determine which years they met (or didn’t meet) their survey participation goals. An overlapping column chart served as a great way to for project managers to determine just that in a matter of seconds.

Ready to make one yourself? Awesome – let’s do it!

To get started, select your data insert a 2D Clustered Column Chart.

Excel, we love you so, but you do some weird stuff. To fix the data, right click and choose select data. Go ahead and delete the year series (oh yes, we’re going to delete lots of things!), select Goal and notice the x-axis is empty…go ahead and click this button and highlight the four years. Voila! Your Goal Series is now included, and you should have two columns in your Excel window.

Next, let’s get these columns on top of one another. To do that, we are going to right-click the Actual Column, select Format Data Series (get familiar with this area of Excel – it’s crucial to a lot of your changes!), and change the axis from Primary to Secondary. The column you want on top is the secondary…and the column you want on the bottom is the primary.

Now that we have these on top of one another, let’s adjust the gap of the Goal column. You can play with the settings to make it look right but I’d say at least down to 75%.

To start to clean this up (it’s still confusing right now!), let’s right-click the Actual column (Excel should allow you to select all of them) and Add Data Labels.

From here on, it’s really turning your chart from a Basic to Bomb Chart (check out this example of how to make yours look awesome). You want to pay special attention to fonts (both the type and size), colors, unnecessary noise (yes grid lines, I’m talking about YOU), and, of course the title! It’s here where you want to leverage data visualization best practices to really get your reader’s attention.

After you’ve made some simple changes, your overlapping column chart it should look something like this:

One thing you might notice is we don’t know what the goal was from looking at the chart – and that’s OKAY. This is really intended to give high-level insight. In other words, was the goal achieved or not? Whether this is as much information as your exec team needs, or you want to create a dialogue, I highly suggest this minimalistic chart for easy actual-to-goal comparisons!

Want to know how to do this in Tableau? Tune in next time and don’t forget to check out my posts on how to Getting Started with Tableau.  

Research on Evaluation: It Takes a Village (The Solution)

Our first post lamented the poor response rates in research on evaluation. There are many reasons for these poor response rates, but there are also many things that we can do to improve response rates and subsequently improve the state of research on evaluation.

How can evaluators improve response rates?

Coryn et. al (2016) suggests that evaluators find research on evaluation important. However, the response rates to these projects would suggest otherwise. As with any area of opportunity, there is often several components that influence success. Yes, evaluators should naturally care more about propelling our field forward, but the ability to change that without amending our practices as researchers seems unlikely. Therefore, we believe that the importance of participation must be built and to do we need to focus on what evaluators see as valuable research. Researchers must also take care to carry-out research with sound methodologies. Some recommendations for improving response rates as evaluators include:

  1. Conducting research that is relevant to the field of evaluation while maintaining a high standard of rigor. You can increase the likelihood of this by…
    1. Piloting your study (grad students and colleagues are great for this!)
    2. Asking for feedback from a critical friend
    3. Having evaluation practice guide or inform the research questions
  2. Reduce the cognitive load on participants by making our surveys shorter and easier to complete. You can do this by tying your questions to your research questions. It’s fun to have lots of data but it is even better to have meaningful data (i.e. stop asking unnecessary questions).
  3. Apply Dillman’s Tailored Design method. This includes things like:
    1. Increasing the benefits of participation, such as by asking for help from participants or providing incentives for participation
    2. Decreasing the costs of participation, such as by ensuring no requests are personal or sensitive in nature and that it is convenient for participants to respond

What can the AEA Research Request Task Force do?

The AEA Research Request Task Force is also a crucial component of this process, acting not only as a gatekeeper to the listserv, but also as quality and relevance control. Currently, samples of usually 1,000-2,000 evaluators are sent out for every research request. If we could increase the response rate, we could decrease our random sample and decrease the load on the AEA membership. Some recommendations for new policies for the task force include:

  1. Policies that would satisfy Dillman’s Tailored Design Method, including allowing:
    1. Personalized contact (e.g., providing names to researchers)
    2. Repeated contact to participants
    3. Contact via postal or telephone
  2. Consider sending out survey requests themselves to improve the legitimacy of survey requests and reduce confidentiality concerns
  3. Have more stringent rigor and relevancy standards to decrease the likelihood that participating evaluators get frustrated over the surveys that sent out and subsequently opt out of future research

Conclusions

We believe that evaluators should care more about the importance of research on evaluation and that it should be more visible in the field so that practitioners know about it and how it can improve their practices. However, it is our responsibility to improve our field by being good research participants. So please, if you ever receive a request to participate in a research on evaluation study, please do so. You are helping our field of evaluation

Collaboration is Awesome

ASR_9983-600x600

This post was written in collaboration with Dana Linnell Wanzer. Dana is an evaluation consultant specializing in programs serving children and youth. She loves Twitter, research on evaluation, and youth program evaluation. If you haven’t already, check out her blog — you’ll be glad ya did!