JUDY MORTON: Before we get going, could I just get a quick little snapshot?  How many of you have some basic knowledge of Malcolm Baldrige quality of work criteria?  Okay so it looks like we have 20, 25 percent. Of those, how many of you have had experience actually applying the criteria with any organization that you've worked with. Great.

So we have a couple experts right here in the group. That will help me. As Colleen said, what she had asked me to talk about is the Baldrige framework as one method for looking at goals and measurement. And what I really like about the, well there's lots I like about the Baldrige criteria. But I think mostly what it's about is creating excellence. And encouraging those of us who work in organizations to get really clear about what success means, what excellence means. Now, Morris here is a study in the lower right hand corner of the big picture. And again, what Baldrige, I think, helps us do is step back a little bit and really look at the big picture and I'd like to take a few minutes to share with you some of the kinds of things that the Baldrige criteria suggests to us. The Baldrige criteria, basically has two different parts. Number one, it has a set of criteria that essentially describe key elements that are needed to optimize organizational performance. The results section of the category actually describe what are the criteria for performance excellence for any particular organization that the criteria would be focused on. Another thing that's really important in terms of sort of the intellectual foundations of the framework it that it's based on a certain set of values which are very explicit. And I think from what I've learned about the values of this group, I think very consistent. Now this is a picture of what the Baldrige criteria look like and I'm going to go into quite a bit of detail here before going further. You'll see on the top there's a umbrella if you will, that describes, it says that it's important for any organization that's considering using the Baldrige to be really clear about, as Colleen said, their mission and their vision and understanding of their values, their, what the environment are is for that organization, their relationships and challenges. Now, this model can apply to any size organization.

The very first applicant that I had an opportunity to review as an examiner was a dental practice. It was a dental practice of about 20 people that were using this, criteria to assist them in getting really clear about what it was they were trying to achieve together, and then describing very specific processes that they were going to use to achieve those results. So it can be used for an organization or an entity as small as a dental practice or it can be used for example, one of the most recent Malcolm Baldrige quality award winners in healthcare was SSM healthcare system out of St. Louis, a 21-hospital system. So it could go from that size to whatever. It also can be applied in various sectors. There are criteria specifically for healthcare and for education and also for business. And it's applied also and there are specific awards if you will, it manufacturing and service and for small organizations as well as for in healthcare.

Today what I'd like to do is focus essentially on one aspect of the criteria, the results section, which relates to organizational performance. And you'll notice those little numbers in the corner there. When an organization, if they so choose to do a self assessment and put in an application for a Malcolm Baldrige quality award, they potentially when they are reviewed and assessed have an opportunity to earn up to a thousand points. And the little numbers in the corner represent how many points each of those different aspects of the criteria represent. And as you can see results, the achievement of success represents almost half of the score. Now, all of these other six categories relate what Colleen referred to as inputs. It's what are key processes that when very well executed individually, but also integrated and aligned with each other, all move together toward creating those results. So any organization that may choose to apply for a Baldrige award would be asked in a 50-page application to describe what is their leadership system. How do their senior leaders communicate the mission vision values to every single person in the organization?  Another, how do they essentially and regularly review performance, so that they can continue to improve it. They also would be asked to describe how do they conduct strategic planning. How do they decide what their long term goals are, what their short term goals are and how do they deploy that strategy to make sure that every single person in the organization understands their role in contributing to the ultimate success of the organization. They also would be asked to describe what are their methods for listening to and learning from their customers or their key stakeholders.

Now in the education arena, the focus is really more on the students as learners for educational organizations that have younger student, often times the family would be included in that customer group. For healthcare organizations the focus would be more on patients and their families. And you organizations would be asked to describe how do they know how satisfied their customers are whether they'd by patients or students. How confident are they in carrying out, for example, healthcare regiments, et cetera. There's also in a, sort of a foundation for all of this has to do with the processes for measuring and analyzing results and then sharing information across the organization. So that's another key aspect of what a well, organized system would do that would be able to contribute to organizational results. There is also a huge section relating to what are the processes and systems used to support faculty and staff who are the providers of care or service. What kinds of recruitment and retention systems are in place?  What kinds of learning systems are in place that support people in furthering their ability to provide service to the key customer groups of the organization.

And last but not least is what are the key processes for improving performance over time. Do they have a consistent performance improvement model?  A planned due study check Act. Do they have, are they using six sigma, or whatever, and how do they systematically select where they're going to improve. What are their key processes that need to improve?  For example, curriculum development, how do they actually improve the curriculum development process?  So those six areas are very important in contributing to results. What I'm going to spend most of my time talking with you about today is the results part, which relates to the measurement and framework. And Baldrige suggests that there are several parts, it's a multi faceted thing. But before that I would like to just briefly talk about the criteria. As I mentioned, it's seven categories, each with different weights. But the emphasis really is on results. I think one thing that's very important about the criteria is to recognize that it's very non prescriptive and adaptable. That there is nothing one must do in terms of measures for example. The choice is really up to the organization to decide for that organization what really is a definition of success. The criteria are updated on a yearly basis. So every year, those criteria based on what's been learned the prior year, are updated. And in fact, I have with me a couple I think I have a total of four. I'm going to give you a website address where you can get these I think, for free. But I have three educate, no two educational and one healthcare. So whoever gets here first can have these. And you're welcome to just by describe the criteria et cetera. And then I think the important thing and the really helpful thing about the criteria is it really represents a way of assessing how are you doing over time.

One of the things that the organizations who essentially use the criteria, although many of them are interested in achieving the award, the important thing that people who have been identified as winners, many of them have applied for five or six or seven years, that they so appreciate is the opportunity to get feedback. It represents a way to assess how are they doing on the road toward excellence. The core values, and this is one of the, the things that is different from your slides, is there's a set of core values. They look a little bit different in healthcare and education. When I first put this presentation together I focused more on the education side and at the encouragement of Dr. Redick decided that we should add healthcare too. Recognizing that with this group so much of your work also for many of you has to do with clinical things. And so we've included the healthcare criteria as well. But as you can see, the values have to do with visionary leadership, learning management by fact, it's very important. A systems perspective looking at the whole, focus on the future, management for innovation. The way they differ slightly with the education criteria versus the healthcare criteria is for education is more focused on learning centered education where as with healthcare obviously it would be focused more on patience centered excellence.

Social responsibility in education is identified as a value where as community health is added with thee healthcare values. And then faculty, are an additional group or state called a group and education beyond what you would see in a healthcare organization. So you may be wondering, who was this Malcolm Baldrige guy anyway?  Interesting, is Laura still here, Laura, he was a fed. He was the 26^th Secretary of Commerce, apparently was an incredibly excellent manager and took his organization, Scofield, to great heights, did some amazing work when he was at the Commerce Department, very colorful and beloved, he died actually in a rodeo accident. But he had lots of interests, kind of a renaissance guy. Anyway, this was very much of a passion for him. He was very, very interested in essentially creating this award that would recognize excellence, would really provide a way to share best practices with, across the country, essentially very interested in promoting America's competitiveness. The award itself recognizes excellence and is awarded every year, only to organizations who actually achieve incredibly high levels of performance in the variety of measurement areas that we'll be talking about in a moment. I think the most important thing about why this is important is that when this model is rigorously applied, it does produce documented world class results in a variety of sectors. And as Colleen suggested, it may have some applicability here. Just to give you a little background in terms of organizations that have been named National Malcolm Baldrige Quality Award winners. So far there have been four in education. Actually it was several years that the healthcare, or the education criteria existed before any winners were identified because it was the judges determined that the level of performance was not at the level it needed to be before there could be this level of recognition.

As you can see here, there are four different kinds of educational institutions that have been recognized. Just as example of a result. Chugach School District is a very interesting school system up in Alaska that serves residents of the Kodiak Islands and they're all spread across the islands. This school went from and I don't have the numbers exactly straight in my head, but it's something like 42 percent of all of their students graduating to over 90 percent of their students graduating. And they did some phenomenal work to create that kind of environment among their faculty and staff and partnering with families to create that. But that's the level of performance that we're seeing with these kinds of winners. Healthcare, these are the three winners so far that have been named in healthcare. Again, the same situation, the healthcare criteria existed for a long time before any healthcare winners were announced. The first year the first winner was SSM Healthcare out of St. Louis. And just this last year, two winners, Baptist Hospital out of Pensacola Florida and St. Luke's Medical Center out of Kansas City. An example of a kind of result that you'll see for example, from Baptist Hospital is that is a hospital that has achieved 99 percentile patient satisfaction ratings for inpatients, outpatients, emergency department patients, for four to six years running depending. They also have their staff satisfaction scores are higher than any their vendor has ever identified in with any organizations they have ever surveyed and physician satisfaction ratings off the charts. They've really done a lot of focus and thinking about how to create an incredible service culture. So as this person said, well one day I do have this recurring dream that we may someday see results.

It's now time to talk about organizational performance. And I'd like to just share with you the model that Baldrige suggested organizations consider as they think about identifying what are their key results areas. The results categories look a little different in education and healthcare but they're pretty parallel. And they're all equally weighted. First set of results in education has to do with, and a lot of the kinds of things you were talking about yesterday is what our, essentially how do you know that your students are learning. That means you have to know what you want them to learn and then have a way of assessing how they're learning. In healthcare, looking at healthcare results and I'm going to give you some examples of these later. In education the kinds of results have to do, a second category is student and stay colder focused results. So those would have to do with things like how satisfied are students with the level of instruction and support they are getting, and any other stakeholders that are related. Same thing for patients, how satisfied are patients and families with the care and service that they are receiving.

Another whole category relates to budgetary and financial performance as well as market performance. So for example, if you were from a hospital like I am, we would be looking at market share, perhaps as segmented by different service lines. We'd also be looking at budget performance. In a grant, funded entity, you may be looking at different things. Or you may be looking at things like cost per student. Another whole key, very, very important results areas relates to faculty and staff, in healthcare staff and the work system. And I'll give you some examples of that. And then the last two categories, and I'm going to go into much more detail for each of these. One relates to organizational effectiveness. How effective are you being, both in terms of some internal processes that create those desired results as well as potentially some larger more global measures?  As well as measures of governance and social responsibility. I might mention that governance is an area that has been identified as an area of increased focus with the Baldrige just in the last couple of years, given some of the recent events in the United States. So in terms keep considerations around measurement and one of the things that Baldrige looks at and we as examiners are trained to look at when we're looking at reports of results over time, is we're looking for measures that are actually reliable and valid, hopefully proved. So for example, leadership index that we heard about yesterday, if there's been some validation of that, that would be important. Or is you're using a patient satisfaction survey that you know that it is reliable and valid instrument.

One thing that's very key is performance over time. And I know as we'll be talking later today about some measurement kinds of issues with MCH, looking at performance over time is very important. What levels of improvement are you seeing, maybe between a first year student class and a second year, looking at cohorts for example?  Or also looking at other kinds of measures in terms of are you seeing the lines move in a positive direction, hopefully?  The other thing that's very important and helpful again from an improvement perspective has to do with, and all applicants are asked not just to describe what their measures and show what their performance looks like over time, but also show how does their performance compare to other organizations that have similar processes or outcomes. So they can take a look at, for example, with patient satisfaction you could be able you could be improving slightly over time but it helps you to know how well you're doing also if you can see how does your performance compare to other organizations that are providing similar services. What that also helps do is if you identify the potential you may not be performing perhaps you're at the 50^th percentile and you know that it is possible to get to the 90^th percentile, there may be some learning partners that you could work with who could help you figure out how to get to the 99^th percentile. Or visa versa, many organizations for example, SSM that has 21 hospitals. All of their entity share common metrics and they share with each other and they can identify where there's variation. So those parts of the organization that are performing particularly well in one area can share what they have learned and what kinds of processes they have that are creating those and visa versa, they can learn from those who are achieving in a, at a higher level in another. And Baldrige does encourage you to take a look at, how does your performance compare to others, how does it compare to averaging, in particular how does it compare to best in class, sort of suggesting and encouraging us to think about excellence.

The other thing that's strongly encouraged is taking a look at segmented data. Again, that represents opportunities for improvement. So for example, if you look at any kind of student outcome measure whether it be instructional gains or participation in advocacy, in your case or whatever. Is there variation by ethnicity, by age, be gender or whatever and to see if there are variations then what could be done in terms if you want to move that measure. So what I've done here, and this is another place that some of the slides are different because after I prepared the original slides, I had an opportunity and got access to a few more MCH measures and ideas. Up on top, what you'll see are, here's some sample measures of student learning and I'm not going to go over them in great detail, but things like student retention rates, percent of fully licensed graduates within specified time periods. I know a dental school application I looked at used as an outcome measure on a national board results, they looked both at passage rates as well as average scores over time. On the bottom here are some examples of kinds of metrics that have been used and actually Julie McDougal and Gregg Redding had done a really nice paper on outcomes of theirs.

These are examples of some of the kinds of metrics here and some things that were suggested from the MCH Strategic Plan. Sample measure in healthcare, these are all kinds of measures that have been used by some of the National Malcolm Baldrige Healthcare Quality Award winners. Things like mortality rates, pneumonia rates, no socomial infection rates, obviously with in your setting where you're offering very specialized clinical services, these would need to be adapted to the various clinical services that are being provided. And again, taking a look at those outcomes and segmenting them perhaps be socio economic class, gender, ethnicity et cetera. Here's some samples in that second measurement category having to do with the student stakeholder or the patient stakeholder. Issues having to do with student satisfaction perhaps with instruction, as Colleen suggested, looking at results over time also compared to others and also differences among sub groups, which may represent. I know one education applicant that I took a look at, they did a lot of segmentation by sub groups and identified students that were struggling and they really were able to provide some more remedial instruction and move their levels of performance to much higher levels because they were able to identify folks who needed help and to work with them.

Patient results, some examples that are used by Baldrige winners are patient satisfaction levels as well as things like compliment to complaint ratios. What I want to point out about all of these is these are just examples and every organization, what Baldrige suggests, needs to pick for them, which are the measures of success. Here's some sample measures in the budgetary financial market areas. Things like operating cost per student, how much state or federal revenue per year, budget performance, student loan, default rates. In organizations that are in a more competitive market, market share, or external ratings of market excellence or you could even look at things like public and private funding for research training or whatever. Here's some sample measures for that fourth category, faculty and staff results. Now you will notice many measures that are similar and will be very familiar to those of you in academic settings around publications, you can look at student faculty ratios, professional development expenditures per person, you could look at turnover rates, percent of faculty with different kinds of educational levels, safety issues, salary survey results, labor grievance results, so there's a variety of different kinds of things that could be looked at.

Measures of organizational effectiveness, is a really flexible kind of category. The kinds of things that often times are looked at are things like employer ratings of the preparedness of graduates, thinking about potential employers as potential customers, student job placement rates, percent of graduates in healthcare leadership positions within X years of graduation. The other kind of measure that's looked at is key internal measures that are key processes that promote the effectiveness of the program. So for example, the effectiveness of the computer support for students, different measures of that whether it be the percent of time or the up time of the computer, purchase order turn around, et cetera. And then external awards is often times something that is identified in this category or another measure might be percent of (inaudible) who say they would go through the program again if they had the opportunity. And last but not least governance and social responsibility.

Percent of students, faculty, or staff providing services to the community, United Way contributions, people participating in advocacy. So again, those are all samples. So I hope with that brief little measure, this is one of my favorites. I have a lot of hats to wear in my job and this is one of my favorites. So hopefully, you'll think of measurement as something that is actually, can be kind of fun and really selecting what are those key things that really are indicative of success for us?  I wanted to just give you, and this is in the slide that you have. There is information here about how you can get more information about the Baldrige criteria, their website, you can get copies of the criteria there. Other ways to learn more, both, many states have their own quality award. They often offer workshops. Also, Baldrige offers many workshops across the country. I know Colleen had an opportunity to go to one and found it to be quite inspiring. Another thing you may want to consider and I've talked to a couple of people about this, you may want to think about coming an examiner. It's a, it is a huge volunteer job, I can tell you it takes a lot of time. I can also tell you that for myself, it's been one of the most rewarding professional development things I've ever done for myself. It's been a fabulous opportunity to learn a lot about what makes really excellent organizations even more excellent. And I'd be happy to talk with any of you about that experience if you like. You may want to think about completing an organizational self-assessment, not easy. It's a fifty-page self-assessment. But if you then throw it in as an application, you get a 50 page report back that gives you essentially very specific ideas about what are the strengths of the program and opportunities for improvement. You may want to even think about applying for the award.

So you're going to have a chance to flex your measurement muscles later and when you select measures, we just encourage you to think about the goals of the national training program and I know that that program, that document has recently been circulated. I think it's important to take a look at the balance. And I think a lot of what the Baldrige measurement criteria are about, have to do with balance of both process and outcome measures, looking at different customers, think for your group, are there certain things that would cut across all of your programs and certain things that are more unique for each of the programs. Think about performance over time and compared to others. And then balance between multiple and competing systems. There are many of you operate in an academic world and yet the training program may have different kinds of goals that are more community based and so thinking about those things as well. So thank you very much. It was fun to have an opportunity to share.