Research on Evaluation: It Takes a Village (The Solution)

Our first post lamented the poor response rates in research on evaluation. There are many reasons for these poor response rates, but there are also many things that we can do to improve response rates and subsequently improve the state of research on evaluation.

How can evaluators improve response rates?

Coryn et. al (2016) suggests that evaluators find research on evaluation important. However, the response rates to these projects would suggest otherwise. As with any area of opportunity, there is often several components that influence success. Yes, evaluators should naturally care more about propelling our field forward, but the ability to change that without amending our practices as researchers seems unlikely. Therefore, we believe that the importance of participation must be built and to do we need to focus on what evaluators see as valuable research. Researchers must also take care to carry-out research with sound methodologies. Some recommendations for improving response rates as evaluators include:

  1. Conducting research that is relevant to the field of evaluation while maintaining a high standard of rigor. You can increase the likelihood of this by…
    1. Piloting your study (grad students and colleagues are great for this!)
    2. Asking for feedback from a critical friend
    3. Having evaluation practice guide or inform the research questions
  2. Reduce the cognitive load on participants by making our surveys shorter and easier to complete. You can do this by tying your questions to your research questions. It’s fun to have lots of data but it is even better to have meaningful data (i.e. stop asking unnecessary questions).
  3. Apply Dillman’s Tailored Design method. This includes things like:
    1. Increasing the benefits of participation, such as by asking for help from participants or providing incentives for participation
    2. Decreasing the costs of participation, such as by ensuring no requests are personal or sensitive in nature and that it is convenient for participants to respond

What can the AEA Research Request Task Force do?

The AEA Research Request Task Force is also a crucial component of this process, acting not only as a gatekeeper to the listserv, but also as quality and relevance control. Currently, samples of usually 1,000-2,000 evaluators are sent out for every research request. If we could increase the response rate, we could decrease our random sample and decrease the load on the AEA membership. Some recommendations for new policies for the task force include:

  1. Policies that would satisfy Dillman’s Tailored Design Method, including allowing:
    1. Personalized contact (e.g., providing names to researchers)
    2. Repeated contact to participants
    3. Contact via postal or telephone
  2. Consider sending out survey requests themselves to improve the legitimacy of survey requests and reduce confidentiality concerns
  3. Have more stringent rigor and relevancy standards to decrease the likelihood that participating evaluators get frustrated over the surveys that sent out and subsequently opt out of future research

Conclusions

We believe that evaluators should care more about the importance of research on evaluation and that it should be more visible in the field so that practitioners know about it and how it can improve their practices. However, it is our responsibility to improve our field by being good research participants. So please, if you ever receive a request to participate in a research on evaluation study, please do so. You are helping our field of evaluation

Collaboration is Awesome

ASR_9983-600x600

This post was written in collaboration with Dana Linnell Wanzer. Dana is an evaluation consultant specializing in programs serving children and youth. She loves Twitter, research on evaluation, and youth program evaluation. If you haven’t already, check out her blog — you’ll be glad ya did!

Research on Evaluation: It Takes a Village (The Problem)

 

Response rates from evaluators are poor. Despite research suggesting that AEA members consider research on evaluation as important, response rates for research on evaluation studies are often only between 10-30%.1

As evaluators ourselves, we understand how busy we can be. However, we believe that evaluators should spend more time contributing to these studies. These studies can be thought of as evaluations of our field, such as: what our current practices are, how should we train evaluators, what can we improve, how do our evaluations lead to social betterment, and more are just some of the broad questions these studies aim to answer. These studies can also help inform AEA efforts on the evaluation guiding principles and evaluator competencies.

Why are we seeing poor response rates?

  1. Response rates in general are poor. Across the world, response rates are declining. We are not unique in this regard. This phenomenon is happening in telephonemailing, and internet surveys alike.
  2. Poorly constructed surveys. Unfortunately, some of this issue is probably within researchers themselves. They develop surveys that are too long or too confusing so evaluators drop out early from the study. For instance, Dana’s thesis had a 27% response rate but only 59% of participating evaluators finished the entire survey, which took participants a median 27 minutes to complete. To improve response and completion rates, a more succinct survey would have worked better.
  3. Evaluation anxiety. We often think about evaluation anxiety in our clients, but these research on evaluation studies flip the focus to ourselves. It may be anxiety-provoking for evaluators to introspect—or let other evaluators inspect—their own practices. As an example, participants in Deven’s research on UFE were asked to describe their approach to evaluation after selecting which “known” approaches they apply. Some participants explained that they did not know the formal name for their approach, or they just chose the one that sounded right. This could have been anxiety-provoking for participants and reduced their likelihood of participating or completion the study.
  4. Apathy. Perhaps evaluators just do not care about research on evaluation. Many evaluators “fall into” evaluation rather than joining the field intentionally. They may not have the research background to care enough about “research karma.”
  5. Inabilities to truly use Dillman’s principles. If you know anything about survey design, you know about the survey guru Don Dillman and his Tailored Design Method for survey development. Some of the methods they recommend for increasing response rates are to personalize surveys (e.g., use first and last names), use multiple forms of communication (e.g., send out a postcard as well as an email with the survey), and repeated contact (e.g., an introductory email, the main survey email, and multiple follow-ups). However, these methods are unable to be used with AEA members. The research request task force does not provide names or mailing addresses to those who request a sample of evaluators and they limit contact to members to no more than 3 notifications over no more than a 30 day period. This makes the tailored design method difficult to implement.

Our next post will discuss what can be done by evaluators and the AEA research task force to improve response rates.

ASR_9983-600x600

This post was written in collaboration with Dana Linnell Wanzer. Dana is an evaluation consultant specializing in programs serving children and youth. She loves Twitter, research on evaluation, and youth program evaluation. If you haven’t already, check out her blog — you’ll be glad ya did!

Footnotes

  1. Notably, the study on research on evaluation had a response rate of 44% (Coryn et al., 2016). While this is much higher than most research on evaluation studies—and it is unclear how they achieved this since all they mention is they used Dillman’s principles—it is still low enough to call into question the generalizability of the findings. For instance, it may be more accurate to say only 44% of evaluators care about research on evaluation since the remaining 56% didn’t even both to participate!

For consultants and consultants-to-be: BONUS POST!

rita-morais-108397

BONUS POST WITH DR. GAIL BARRINGTON!

And you thought the fun was over…

Whether you are thinking about venturing out on your own or have already started, this series will arm you with advice from seasoned consultants! This post features Dr. Barrington, who provides her insight on common questions consultants (or consultants-to-be) might have. Be sure to check out the first and second part of this series!

Bio:

Gail Vallance Barrington is a graduate of McGill University (BA) and Carleton University (MA) and holds a Doctorate in Educational Administration from the University of Alberta (1981). She is a Credentialed Evaluator and a certified teacher. In 2014, she was made a Fellow of the Certified Management Consultants of Canada. Since starting her consulting practice in 1985 she has conducted over 130 program evaluation studies in the fields of education, health, and research. Her top-rated book, Consulting Start-up and Management: A Guide for Evaluators and Applied Researchers (SAGE, 2012) continues to be popular. In 2008 she received the Canadian Evaluation Society award for her Contribution to Evaluation in Canada and in 2016 was honoured to receive the American Evaluation Association Alva and Gunnar Myrdal Award for Evaluation Practice. She teaches courses in qualitative research and program evaluation for several universities and provides webinars and workshops on consulting skills.

Gail Barrington closeup 2016

What made you decide to become a consultant?

I began to work on program evaluation contracts while teaching as a sessional instructor at the University of Calgary in the Faculty of Education. For a while I did both. Then a term came along when, due to program reorganization, the courses I was teaching would not be offered. I had to decide what to do, wait for a year with half-time consulting or go at it full time. This was not an easy decision as I had always worked for a school, college, or university, never responsible for my own pay check. Going out on my own was very scary. I can remember sitting in my car looking across a rainy street at the office building I had selected and wondering if I could really work there. I took a big breath, got out of the car and the rest is history. My consulting business opened on November 1, 1985.

Could you share some best practices for remote consultancy?

I often work with clients at a distance. For example, I worked for many years on a national evaluation project for the federal government in Ottawa, Ontario while I lived in Calgary, Alberta. I made a point of holding quarterly meetings in person because I felt that being on site with the client was necessary to get a sense of their work, their issues, and their reaction to what we were doing. Email and Skype are fine but in, the end, it is personal chemistry and partnership which bond a project together. In that project we spent time together informally as well as during our day-long meetings. We had fun skating, sailing, and going out to restaurants. These informal activities strengthened our understanding of each other and we accomplished a remarkable amount of success as a result.

Do you suggest that consultants market themselves as a specialist or a generalist?

I think you need a specialty area and then you can branch out from there. New consultants should focus both on what they like doing best and on what they have received good feedback about. Your specialty area will expand and morph over time but having a clearly defined area of specialization as a foundation is a great way to start.

How do you measure the success of a project?

Success is not measured by the final report. It is really measured by the extent to which positive program change occurs. Evaluation is all about making a difference through informed decision making and of course our role is providing the evidence needed. The real results happen long after the report is finished, and we have gone on our way. Not often enough do we circle back to find out what happened afterwards.

Finally, what advice do you wish someone had given you as a new consultant?

Hang in there and find some good colleagues. When I started my business, I didn’t have any role models and so I had to make it up as I went along. The closest approximation I could find to the independent evaluation consultant was the independent business consultant. As a result, I became a Certified Management Consultant (CMC) so I could have some colleagues to talk to. However, research was not their interest area and so I was still struggling about how to conduct evaluation research and bill for it gracefully. Happily, I found the American Evaluation Association (AEA) Independent Consulting Topical Interest Group (IC TIG) and there at last were business people with a social justice perspective. We continue talking to this day!

Want more Dr. Barrington? Visit her website!

For consultants and consultants-to-be: expert advice (pt. 2)

rita-morais-108397
It’s here — part two! 

Whether you are thinking about venturing out on your own or have already started, this two-part series will arm you with advice from seasoned consultants! This post features Ann K. Emery, who provides her insight on common questions consultants (or consultants-to-be) might have. Check out the first post in this series here.

Bio:

Ann K. Emery is a speaker, workshop facilitator, and blogger, passionate about “making technical information easier to understand for non-technical audiences.” In other words, she is a dataviz expert! Ann is also a well-known blogger, bringing practical tips to those looking to transform their data into effective stories through the use of data visualization. Without further ado, let’s dig into her tips! 

IMG_9880

How did you prepare for running your own consulting firm?

Launching my own consulting firm was a happy accident. I started blogging back in 2012, just for the joy of sharing skills with others, without expecting it to lead anything. And I always enjoyed public speaking and leading workshops. My name got out there. People read something I wrote or saw me speak at a conference. I started getting a few invitations to give talks and redesign the visuals in reports. And then I got a few more invitations. And a few more. At the time, I was working full-time and doing grad school at night. I had limited bandwidth for independent consulting projects. In Spring 2014, I finished grad school and had the time to accept some side projects. I did the math; the projects would actually pay more than my current (good) salary. People started asking when I was going independent. I hadn’t considered going solo. I had planned to stay at my current position for a long time. Over the summer of 2014, I spoke with a dozen of my mentors. I got great advice:

“Don’t even think about quitting your salaried job until you have a years’ worth of household expenses saved—and be willing to lose every penny if you’re not profitable the first year.”

“You’ll work harder than ever, but the work will be more fulfilling than ever.”

In the fall of 2014, I was having dinner with some girlfriends, and mentioned that I might go solo someday. “Well, what are you waiting for?” one asked. I didn’t have a good answer. That next week, I put in my notice.

For those wondering how about the transition to a consultant, did you continue working at a 9-5 job until you became established?

I’ve met two types of consultants: those who find themselves with spare time (job loss, just finished grad school and they’re job hunting, etc.) and those who have already built a reputation for doing great work and have prospective clients banging down their door. The first type struggles to take in work. The second type struggles to turn it down. The second type has no choice. You have to quit your salaried job and start your own company. You work harder than ever, for a while. Then you get better at subcontracting and saying no. You get to choose which type of consultant you want to be. Pull the trigger too early—before you’re established—and you may always struggle to bring in work and pay the bills.

How do you avoid being spread too thin?

I hire smart and talented subcontractors like you!

More importantly, I say no so that I can say yes. I don’t appear on podcasts (I’m visual so an auditory medium has zero appeal). I don’t write guest blog posts (my clients hire me to write blog posts so it doesn’t make business sense to write for free). I don’t work for free (I like to keep a roof over my head). I have to decline projects that aren’t a perfect fit so that I have creative energy to rock the ones that are.

Describe a time when you dealt with a difficult client (or situation). How did you make things work?

I divide my projects into two broad categories: training and design.

In training projects—my keynotes, workshops, webinars, and individual coaching sessions—I haven’t had difficult clients, but I have had inexperienced clients. The contact person has been put in charge of planning the keynote address for their conference for the very first time. I often need to teach them about coordinating with A/V staff, setting up projectors, connecting and testing the microphones, and so on. Planning a talk of this level can be a stressful experience for my clients. They want the logistics to be perfect. I try to walk them through the unknowns and alleviate as much of the stress as possible. I’ve given a billion talks. I’ve seen all sorts of stage setups. Everything that could go wrong has gone wrong—projector lightbulbs burning out mid-talk, fire drills, laptop batteries dying, malfunctioning microphones. I get migraines a few times a year—the kind where your vision and smell are all messed up—so I knew it was only a matter of time until I got a migraine during a speaking engagement. It happened in February. I could only see a sliver of my slides thanks to tunnel vision. Then the smells and nausea started. I gave participants a coffee break, puked in the bathroom, and came back and finished the talk. From the audience’s perspective, it was one of the better workshops I led all year. Good public speaking is more about rolling with the punches than about careful preparation. I let my client know that I’ve experienced every possible projector and microphone hiccup and that the talk will be stellar no matter what. 

In design projects—revamping existing reports or designing the visualizations for client reports from scratch—I haven’t had difficult clients, but I have had difficult timelines. Contracting takes longer than expected because we need a signature from someone who’s on vacation. I’m graphing the data and notice that the numbers don’t make sense and they have to re-run the analyses to fix a few typos. Every consultant I know has been in this situation: Something goes wrong that’s outside of your control, and you’re the one who has to give up your weekend to fix it. We all notice the red flags early on. In the past, I’ve tried to give the project the benefit of the doubt. This project will be different, I lie to myself. Sure, their timeline is tight, but maybe everything will go according to plan this time.

My number one goal in 2018 is to trust my gut instinct and decline the projects with too-tight timelines.

How do you measure the success of a project?

Repeat clients and referrals!

A few months ago, I gave a mediocre workshop—or so I’d thought. I’d pose a discussion question to the group, and people just stared at me with poker faces. I’d tell a joke, and people just stared at me with poker faces. I couldn’t understand why the workshop structure I’d carefully crafted over the years had fallen flat. I left feeling deflated. Over the weekend, I had serious self-doubt, questioning whether I was even in the right career path. Then, on Monday morning, the client emailed me, praising the workshop and saying it was the best they’d ever attended. They invited me to return to their organization for another few days of workshops. I returned, gave another few days of workshops, and left with the same self-doubt. For the second time, nobody responded to my discussion questions or laughed at my jokes. And then—you guessed it—the client emailed me, praised the workshop, and invited me to return. The organization is accustomed to traditional, buttoned-up lecturers. My skin gets thicker each time, so when I return for the third series of workshops, I’ll be prepared to pose non-discussion discussion questions and tell my unfunny jokes.

In design projects, I used to think that a repeat client was a bad thing. If I redesigned the report well the first time, the client should be able to follow my steps and do it themselves the next time, right? But my clients are often pressed for time. Or, they can get the design 90% of the way there, and they need me to nudge the visualizations to the finish line. I’m working on a multi-year project right now. Each year, my role shifts. At first, I was creating the visualizations myself. Later, I was coaching their staff members through the process, making minor adjustments to their drafts, but creating very little myself. Other consultants have warned me against this approach, worried that I’ll teach clients too much and be out of a job. But my instincts keep telling me that training up staff is a net positive. I’ve taught the staff so much during this multi-year project that I wish I could hire them. Literally. I tried to subcontract part of a project to one of the women, but we discovered that our contracting language wouldn’t allow it. We’ll definitely be working together again someday.

Finally, what advice do you wish someone had given you as a new consultant?

I’ve learned from the best: Herb Baum, Tanya Beer, David Bernstein, Dave Bruns, Isaac Castillo, Stephanie Evergreen, Edith Hawkins, Rodney Hopson, Kylie Hutchinson, Helene Jennings, Cole Knaflic, Chris Lysy, Kevin McNamee, Johanna Morariu, Kim Narcisso, Veena Pankaj, Maryfrances Porter, Jon Schwabish, and Trina Willard.  

I adore each of these people for telling me what I need to hear, not what I want to hear. They’ve given me all the personal and professional advice I’ll ever need. There’s nothing I wish I would’ve known earlier—just advice I wish I would’ve followed earlier.

Want more?!

If you’re interested in learning more from Ann, check out her website. You won’t be sorry, and I bet you’ll be adding it to your favorites

“Expert” Photo by Rita Morais on Unsplash

For the consultants and consultants-to-be: expert advice.

rita-morais-108397

Are you considering consulting?

Whether you are thinking about venturing out on your own or have already started, this two-part series will arm you with advice from seasoned consultants! Dr. Stephanie Evergreen and Ann K. Emery will be featured, providing their insight on common questions consultants (or consultants-to-be) might have. Note: both are heavily focused on data visualization, reporting, and infographic design, but their advice can be applied to a variety of fields! First up, Dr. Evergreen.

Bio:

Dr. Stephanie Evergreen is a “Research nerd turned information designer,” and owner of Evergreen Data. She is a seasoned data visualization blogger, workshop facilitator, and the author of two books, Effective Data Visualization: The Right Chart for the Right Data and Presenting Data Effectively: Communicating Your Findings for Maximum Impact. Dr. Evergreen’s background is in interdisciplinary research, which allows her to bring data visualization and reporting into a variety of settings. Below, she shares her experience of transitioning from a 9-5 and tips for successfully dealing with pricing and difficult clients.

Deven Wisner AEA 2017 Evergreen

How did you prepare for running your own consulting firm?

Mentors are totally necessary. Everyone would benefit from some detailed guidance on the nitty gritty details of running a business. Try Gail Barrington’s workshops (attended a few workshops per year). (If you are interested, here is a link to her AEA estudy). But like most people, I was just winging it and learning as I go.  It’s better to get going than to wait until you think you have it all figured out.

For those wondering how about the transition to a consultant, did you continue working at a 9-5 job until you became established?

“Yes! Most consultants start their side gigs while working a 9-5. The idea is ramp up the consulting as you do that – until you feel you are working two 9-5 jobs.”

Pricing, the tough but critical conversation…

#1 – do not work for free. Even for “exposure.” People can and should pay you for your expertise. Beginning consultants feel like they can’t charge until they get more experience but that’s just your imposter syndrome talking. Start charging.

People see more value in things that cost money. Do not undercut yourself. I made some early mistakes of charging too little and then those clients became repeat clients and they told their friends – which was great!  – except I had a lot of low paying work because I pitched too low to start. Protect yourself by using a contract.

How do you avoid being spread too thin?

– A giant whiteboard.

– Moved everything to the cloud, running notes, calls, planning, documents.

– Hired an assistant (should have done this sooner!).

How do you keep track of your progress during a project in order to ensure you are achieving the client’s objectives?

“It’s all in your head. If you can’t do that, you will not succeed as a consultant otherwise.” But we also use Slack to keep ourselves on track and on schedule. We also build in frequent check-ins with the client to make sure we are matching their vision.

Describe a time when you dealt with a difficult client. How did you make the relationship work?

“I have learned to avoid these clients by watching for red flags during the initial discussions. I actually keep a ‘Red Flag List’ that I continually update when I learn from my mistakes. One of my mentors passed on something her mentor told her: Projects should be ‘Fun, Lucrative, and No Assholes’ and this guidance has done me very well. But from time to time, difficult clients still happen. One recently kept sending my work for review by their Communications Officer – the same person who failed them so badly, that’s why they hired me in the first place. So, I had to say ‘This is why you hired me. This is my area of expertise. Your Communications Officer’s advice flies in the face of accepted best practice and peer-reviewed research.’”

How do you measure the success of a project?

“I usually contact clients 3-6 months after the contract is over to ask how things are going, do they need any momentum push, etc. That’s when I learn whether I was successful. But other times I learn through new ways, such as two clients who won awards for the report I worked with them on.”

Want more?!

If you are interested in learning more from Dr. Evergreen, check out what she has to say about marketing yourself as a public speaker. Also, make your way back here in a couple weeks when Ann Emery shares her advice!

“Expert” Photo by Rita Morais on Unsplash

How to Customize Colors in Tableau

How to Customize Colors in Tableau

Now you’ve got some fonts that aren’t basic, how about some custom colors? I really like matching my visualizations to my report, which is almost always based on the color scheme of the organization I’m working with. Let me tell you though, this can be a bit of a beast, so let’s get started.

First, identify the colors you want to use. I suggest Adobe Color, which will allow you to upload a photo and get HEX codes (along with complementary colors and other fun things). Of course, you might be lucky and already have your HEX codes available.

Next, you’ll need to determine whether you want to add a categorical, sequential, or diverging palette. I’ll show you categorical. You can add the others as needed using the same method.

With your HEX codes in hand, locate your Tableau Repository. Depending on where you installed this, it could take a minute. When you locate it, there will be a Preferences.tps you need to open in a text editing program.

Deven Wisner Tableau Colors 1Deven Wisner Tableau Colors w

Now, get started with code. You want to add the following before copying anything into the text edit.

Deven Wisner Tableau Colors 3

You will need to copy the basic code down and insert your HEX codes. Tableau lists all the sets of you would possibly need here. Copy, paste, and customize.

HERE IS THE TRICKY PART…

Make sure you are using straight quotes. Anything else will cause all sorts of errors. I’ve included examples of a correct and incorrect repository. This is often not a matter of inserting quotes but changing the auto-correction in the program.

Deven Wisner Tableau Colors 4Deven Wisner Tableau Colors 5.png

Save and close your text editor and open Tableau! To test your colors, open up a sample dataset. On a worksheet, drag a dimension onto the Color Mark. You’ll immediately see those basic default colors but don’t worry – we’re fixing that!

Deven Wisner Tableau Colors 6

Select the Color Mark and click Edit Colors. You should see yours at the bottom!

Deven Wisner Tableau Colors 7

Select your color and be sure to click Assign, otherwise nothing will change. Close the editor and check out your new colors!

Deven Wisner Tableau Colors 8

SUCCESS!

Deven Wisner Tableau Colors 9

If you followed along, you just completed some serious steps in customizing your Tableau visualizations. I call it a small but mighty change. You can go forth knowing that your visualizations are more customized and specific to your project!

P.S. This is cross-posted on Ann K. Emery’s blog (because awesome people stick together)!

How to Customize Fonts in Tableau

Giving Your Tableau Visualizations a Makeover: Custom Fonts

Whether you are using Tableau, Excel, or any other visualization tool, you will come across defaults. The problem with using something right out of the can is that it often does not speak directly to your audience. Your investment in a visualization speaks to your investment in conveying the story of your data.

By now, you may know tips and tricks to de-clutter and minimize the amount of ‘out-of-the-box’ stuff in your Excel charts, but I want to share how to tackle the same problem in Tableau. It doesn’t take long to spot a Tableau visualization. Is that a bad thing? It depends (does that give anyone else grad school flashbacks?). Instead of being recognized for the font (Tableau Book) and colors that Tableau pumps out, have your Tableau visualizations recognized for their utility and story.

I’m going to share how to customize your fonts and colors within Tableau. And, make your life a lot easier (i.e. not changing one component of the visualization at a time.). Repeat after me: “I am better than the defaults.”

Let’s start with fonts, the easier change of the two.

Instead of having to change your fonts as you go, you can change the entire workbook. If you enjoy clicking on every area of your chart and changing the fonts, go ahead… but, I bet you’d rather spend that time on something else. You will also miss one or two! This way the entire book is the same font. You can adjust sizing as needed!

First, open your workbook.

Go to Format in the top, right-hand corner and click Workbook.

Deven Wisner Tableau Fonts 1

On the right-hand side, you’ll have a formatting pane, which includes fonts, colors, and lines. Choose your font (I suggest a nice sans serif), and you’re ready to rumble!

Deven Wisner Tableau Fonts 2

Check back on 11/27 for How to Customize Colors in Tableau.

P.S. This is cross-posted on Ann K. Emery’s blog (because awesome people stick together)!