Career Centre Evaluation Guide: Project Blog

Guide Now Available!

2011-12-22T17:58:00.000-08:00

We are excited to announce that the new resource we have been creating, Career Centre Evaluation: A Practitioner Guide, is now available.

Many thanks to everyone who has been involved in this project, from reading drafts and providing feedback, to sharing encouraging words about the value of this initiative.

To let others know about this new resource we'll be putting our communication plan into full swing in January, including a presentation at CANNEXUS on Tuesday, January 24, 10-11:30 am. Would love to see you there!

New article of interest in Canadian Psychology

2011-11-16T09:03:00.000-08:00

Our guide uses an evaluation framework created by the CRWG (Canadian Research Working Group for Evidence Based Practice in Career Development). For those of you following their work, if you haven't yet seen it, you may be interested in a new article in Canadian Psychology by one of the CRWG members.

Hiebert, B., Domene, J.F., & Buchanan, M. (2011).The power of multiple methods and evidence sources: raising the profile of Canadian counselling psychology research. Canadian Psychology, 52, 265–275.

I became aware of this article as the authors cite our evaluation guide initiative as an example of a project that uses their framework. Thanks for the mention!

External feedback stage complete - Thank you!

2011-11-14T09:00:00.000-08:00

Thank you to the career centre staff and directors from across Canada who reviewed the draft guide and gave us their feedback. We had great geographic representation - with reviewers from British Columbia, Alberta, Saskatchewan, Ontario, Quebec, Nova Scotia, and Newfoundland. Your feedback has been very helpful.

We are now in the final stages of finishing the guide and we should be launching it publicly later this fall!

New website for evaluation group

2011-05-31T08:02:00.000-07:00

If you are interested in evaluation in career services in Canada, then I'm guessing that you, like I, try to follow the work of the Canadian Research Working Group for Evidence Based Practice in Career Development (or CRWG for short).

I recently noticed that they have a new design for their website, and lots of information about their activities. I thought I'd share this in case you haven't seen it yet. Just click the link on their name above.

Draft site under review

2010-11-09T08:40:00.000-08:00

We have been progressing well over the last several months and now have a draft website created. Thought you might find it interesting to see a glimpse of the homepage and a look at what is covered.

For a sense of the structure and scope of the guide, here is our current Table of Contents:

Introduction and Framework
Evaluating Inputs
Evaluating Processes
Evaluating Outcomes
Case Studies
Reporting and Using Evaluation Results
Putting It All Together

Now, back to editing.

The Value of Career Services

2010-10-13T06:45:00.000-07:00

How do we measure the value of career services? NACE suggests that a few measures in their 2010 Student Survey provide some options. They point to
- likelihood of getting a job. The survey showed that more use - 4 or more times in a semester vs once - was associated with a greater likelihood of a job offer
- salary - more use seems to be associated with higher median salaries
- increased use of career services by freshmen. Their interpretation: "Clearly, freshmen understand that they need to be prepared, and are getting an early start."

These are interesting measures. Great to see they are all positive!

However, a couple thoughts:
- might the higher mediam salaries not be truly related to career services use, but be because users of career services are not distributed evenly across all disciplines and types of work?
- why are freshman using the career services more? Is this required in any programs? Are they being "encouraged" by anyone (eg anxious parents)?

I am absolutely convinced that career services have value. To be fair, these stats are really presented as teasers to encourage readers to get the full survey report, and there may be more information there (I've not read the entire thing). However, reading this short article reinforces for me just how complex determining and measuring value really is.

Your Input Welcomed

2010-07-20T08:58:00.000-07:00

We have been busy working on drafting the initial content of the guide. Our starting points were the evaluation framework from the CRWG, the materials the working group had created together in the past, and the experiences and questions coming from the evaluation activities from each member’s office.

We have a great skeleton, and the emerging content is looking good.

After we have a draft of the guide, we will be doing testing – and getting feedback from other career centres – which will be a fantastic part of the process.

But right now, while we are still early in the writing process, I thought it might be a good time to see if anyone else in the community has any suggestions for topics or questions to cover. If you are following this blog, I’m guessing you not only have some interest in evaluation, but you likely have some thoughts about the kinds of topics, or perhaps questions, that you would like to see addressed.

If you do, and you have a couple of minutes, please consider sharing them with us. You can post a comment to this blog posting, which will let other readers see your ideas, or feel free to email me directly if you would prefer that.

I can’t promise that every question will get answered in the guide (we’re going to cover a lot of ground, but evaluation is so enormous that there will surely be a few things that may not fit), but we will review all input and use it to shape what is covered.

Thanks in advance for your input!

Making the Case for Assessment

2010-06-09T11:14:00.000-07:00

"Running a career services program without conducting assessments of operations and services can be risky." So opens the article, Key Aspects of Career Centre Assessment, in today's NACE Spotlight Online.

The article profiles the thoughts and experiences of Mary Feduccia, career services director at Louisiana State University. It explains their assessment cycle, and some specific assessment activities that they use. In addition, they have posted some of their assessment information on an Assessment webpage for their department. This page is well worth visiting for a detailed look at some real examples of a career centre's assessment activities, including comments on what changes and recommendations came out of the assessments.

Ms. Feduccia's parting thoughts are some I think are easy to agree with:
"It’s so easy to put assessment on the back burner, but career centers need to keep going with it,” Feduccia says. “It takes an entire process and a lot of work to be successful. Thoughtful planning, setting and adhering to deadlines, and following up on identified actions to close the loop are required for the assessment process to be as beneficial as possible.”

An Evaluation Framework

2010-04-26T07:19:00.000-07:00

As we’ve discussed how to wrangle this nebulous notion of “evaluation” we have found it helpful to have a framework to guide our thinking.

Luckily, there is a framework available, one that was designed specifically for career services, and is Canadian.

The Canadian Research Working Group on Evidence-Based Practice in Career Development (CRWG) has created a comprehensive framework for evaluation which is specific to organizations providing career-related services. We have chosen to use their framework to structure our work on evaluation thus far, and we will be using it to structure the evaluation guide that we are creating as well.

The CRWG framework is available online in an article titled "Demonstrating Value: A Draft Framework for Evaluating the Effectiveness of Career Development Interventions," published in the Canadian Journal of Counselling.

Have you used this framework to guide your evaluation planning? If so, how have you used it, and how has it been helpful?

Working with Learning Outcomes

2010-04-12T12:02:00.000-07:00

For any readers who are currently using learning outcomes in their work, I wanted to let you know about an upcoming webinar.

CACEE asked me (Cathy) to present "Learning Outcomes: Taking it to the next level."

I'll share some ideas about setting "good" outcomes, strategies for measuring outcomes, and how to use outcomes in quality improvement planning. And there will also be space for discussion and questions.

Monday April 19, 12-1
register through CACEE

Learning outcomes will be one (of many) topics we will cover in the guide that we are developing. Through offering this webinar I hope to hear more about the kinds of questions, issues, and ideas that other career centres across the country are thinking about when it comes to using learning outcomes. If you aren't able to attend the webinar, but still have ideas/questions about learning outcomes you like to share with our group as we develop the guide, please comment on this post, or send an email directly to me. Thanks!

Project Timeline

2010-01-25T07:00:00.000-08:00

When will the Guide be ready?

The project has a 2 year timeline. Within that, there will be four phases:

• The initial stage will develop the content and the website. Content will be researched and written, tools and case studies will be collected from working group members, and new templates will be drafted.
• During the second phase, the new templates, and the draft content and tools will be tested with staff in the offices of the working group members, in order to get feedback. This feedback will then be reviewed and the draft Guide will be edited.
• The third phase will be External Testing, which will take the draft Guide to three other career offices (most likely one in the West, one in the East, and one in Quebec) to get feedback from centre leaders and staff. The Guide will be edited based on this feedback. The final version of the Guide will be prepared for making it public.
• The final phase will disseminate this new tool to career offices across Canada (and more broadly based on interest).

Timeline At-A-Glance

Content & Web Development Phase:
8 months
January 2010-August 2010

Internal Testing Phase:
8 months
Sept. 2010-April 2011

External Testing Phase:
4 months
May 2011-August 2011

Final Production and Distribution Phase:
4 months
Sept 2011-Dec 2011

An Introduction to the Project

2010-01-21T09:05:00.000-08:00

To give you an introduction to this project, here is brief look at WHY we’re doing this, WHAT we want to create, and WHO is involved.

Why:

There is probably not a need to say a lot about the “why” of the project. Evaluation has become one of the most discussed topics amongst career centre leaders. Interestingly, according to NACE, the National Association of Colleges and Employers representing career centres at post-secondary institutions in the US and Canada, “if one were to point to just one critical issue for career services, accountability would likely be it” (NACE Journal, 2005, p. 28). A survey of their members revealed that two of the top three issues of the future for career services professionals are “measuring the effectiveness of your office’s programs and services” and “demonstrating and validating the value of the career services function at your institution” (NACE Journal, 2005, p. 28).

While there is a lot of interest and some activity in regards to evaluation happening, what we regularly hear from career centre leaders and practitioners (and feel ourselves) is that we require more tools and strategies to guide our evaluation efforts.

What:

The purpose of this project is to create an easy-to-use, comprehensive set of content and tools for career office leaders and staff to use as they strive to improve and expand their evaluation activities in order to better monitor and enhance the quality of career support for clients.

In order to make the Guide easily accessible and affordable, it will be produced as an online site with open access.

Who:

Working Group (The University Career Centre Metrics Working Group):

The group of career centre leaders who have conceptualized this project and will be seeing it through are:

Karen Benzinger, Director, Centre for Career Education, University of Windsor
Kristi Kerford, Director, Career Centre, Trent University
Leslie Lumsden, Director, The Student Success Centre, The University of Western Ontario
Kerry Mahoney, Director, Career Services, University of Waterloo
Yvonne Rodney, Director, Career Centre, University of Toronto

This group has been meeting over the last 5 years to discuss and share approaches to evaluation. While discussions and evaluation activities within the group have been broad-ranging, as a first project to create for sharing with the larger community, the group developed a tool for setting and evaluating learning outcomes. This tool was shared with colleagues, including through a presentation at the CACEE National Conference in 2007. The group is now excited to share other aspects of their work on evaluation, and create new tools, for the Guide.

Project Coordinator:
The Project Coordinator is Cathy Keates, Director of Career Considerations, who has been a member of the Working Group since its inception, formerly within her role as Associate Director of the Career Centre at York University.

Funding:
The project is happening because of funding from The Canadian Education and Research Institute for Counselling, (CERIC), (thank you CERIC!) and contributions from the working group members.

Project Blog Introduction – Career Centre Evaluation: A Practitioners’ Guide

2010-01-21T08:58:00.000-08:00

In Fall 2009 we received approval from CERIC for funding for a two-year project to create an online guide to evaluation for career centres (thank you CERIC!). The Guide will be targeted to university career centres, but we hope it may have applicability in other settings as well.

As we kick-off the official start of the project in January 2010 we decided that having a way to involve interested colleagues in the development and progress of the project could both help inform the development of the Guide, and keep you informed about our progress.

We hope that a project blog will:
• keep interested career centres up-to-date on the progress of this project
• provide a way for interested people to be more engaged in this project
• solicit input, examples, and questions to enrich and inform the project content and direction.

The blog will
• start by giving you an introduction to the project, its objectives and timelines
• share project updates as we progress through the development of the Guide
• share new resources – as we discover new evaluation resources along the way (eg articles, training), we will share them with you here
• ask for your questions and input – we want to make sure the final Guide is targeted to the needs of the people who will use it, so along the way we will pose questions to you, and we welcome comments and input.

The blog is being written by Cathy Keates, project coordinator, on behalf of the project team.

Thank you for visiting our blog. We hope you continue to follow the project updates and submit comments and questions along the way. (You can follow the blog by clicking on "follow" at the top left hand corner of your screen). Looking forward to communicating with you through this blog!