So our goal today is to talk a little bit about what is a narrative feedback, you know, what are the elements of making a great narrative feedback, why it's important, and really making that case for why we should try to work this more into our programs, particularly maybe including it as part of faculty development if that's not a program that you have to help improve feedback for fellows. So we've all had evaluations come back to us from fellow, for fellows that have said something between blank box, blank box, five out of five. Does that sound like familiar to anybody? Okay, perfect. Or maybe, you know, one line that says something along the lines of should read more. This person was lazy. Really great. Again, feeling, does that feel about right for some folks? Okay. While not helpful at all, we would like to help improve that and get fellows to really have some more constructive feedback that they can use in their week-to-week practice. Now we're all familiar with, you know, milestones and general assessments that we use that, you know, range from if you use Med Hub or one of the other kind of, you know, programs about, you know, where the fellow is on a spectrum of needing complete help from their attending all the way to an aspirational, you know, kind of quality. And that's helpful in the long term. That's helpful maybe with, at the six-month CCC kind of programmatic level. It's helpful overall, you know, year by year. But what it's not as helpful for is week by week, month by month. How does the fellow think about where am I today and what do I want to work on? What can I use to work on next week? Particularly as more and more rotations have moved to just being, you know, where attendings are only spending one or two weeks with fellows during their time, they don't necessarily have the same longitudinal experience. Show of hands, does anybody spend more than two weeks on service at a time? Yeah, it's pretty uncommon at this point. How many folks do spend two weeks on service in the ICU or POM? Okay. And then one week. Yeah, the majority. Okay. Thank you for that. So, you know, this is, I think, particularly important in an era where now we do spend a lot less time. I'm going to keep using this slide as well. Is that okay? Okay. So, yeah, no, I'm telling you, they're amazing. So, you know, so that's part of the argument for why I think these are important, right? And there's a general approach to assessment, right? So, ideally, you would sit down with your fellow for whatever amount of time you're going to spend with them at the beginning. Discuss, you know, your goals and expectations with them. Observe them during your time together. Give them verbal feedback, maybe midway through and then at the end. And then also write that down. And it's really those two at the end that we have heard from that, you know, provide challenges for faculty. You know, maybe they've given verbal feedback, but they haven't actually written it down. And, you know, why is that an important feature? So, we'll think a little bit about these comments themselves. So, what am I talking about when I'm saying we want to get fellows better narrative feedback in their evaluations back to you as program leadership and back to the fellows themselves, right? So, there are generally a couple of kind of key features for that. One is that they focus on behaviors, right? So, again, before I made a mention about you might get these descriptions, but they're really a description as the person interpreting them of a personality attribute, you know, this idea that maybe they are disengaged or disinterested or lazy or something like that if it's a negative comment like that. But that's not really an actionable thing for somebody to receive that feedback and do something with it. It's really an actionable thing for you as program leadership to act on, to help that fellow come up with a learning plan or some other kind of program to improve that. With that, they, you know, you want them to be actionable. So, this idea of read more is not as helpful as, you know, a specific topic-based plan and that they should be specific. So, right now, if you were to see on this slide, you see a couple of different examples of things that I've talked about, but I've kind of read them out loud and described for you. So, you know, another example of this is you might have received feedback for a fellow that says, you know, responds well to feedback. I mean, that's great in that it's positive. It's a great attribute to know about your fellow. But what would be more helpful is to say, you know, I observed them during a, you know, procedure, a central line placement. We talked about some feedback around setup. And then I observed them and that next, the next time around, I observed them implementing that, you know, that feedback. Right? That's much more helpful for them to know that, you know, they take feedback well and this is a version of how they engaged with that. So, I think that is a good place to start. So, with that, why don't we then move on to our thinking about our panel. So, that is some of, so those are some of the features of strong narrative feedback. Right? They're actionable. They're specific. Obviously, they're timely. And they focus on behaviors. So, if that's so great, there's often one challenge to this, which is they're not numerical. Right? They're not, you know, they're not a scale. And shouldn't scales be more objective? And we'd like to argue that the answer to that is no. You know, scales are really generally fairly arbitrary. And again, we're thinking through week by week, month by month, how can fellows improve? And scales really do very little for them to help them with that. So, with that, I'd like to start asking some of my panel colleagues about questions. And along the way, I'd love for this to be a conversation with you all. So, please come up at any point to ask a question, to provide feedback, or if we're talking about a specific, you know, challenge that we've seen or a behavior, you know, a kind of programmatic input, and you have a version of that that you'd love to share with the group, just please come up and come to the microphone and share that with the group. So, with that, team. So, I've just made this whole point about why we think narrative feedback is important. But tell me, you know, why do you think it's important to consider emphasizing narrative feedback based on your experiences with your programs and, you know, just experiences with your institutions? And I'll start with Mark. Well, thank you. So, I think one of the things that comes to mind with Hugo's talking about this is we need to have some sort of documentation, right? Everyone's busy. Everybody has a short memory about things that was like the one critical piece of feedback that you ever gave to that fellow. And so, we need to have that written down. And so, I think the first thing to think about is what are strategies that we can employ to get those written down? So, I don't know if I'm alone here, kind of in the dark, but I don't know if I'm alone here, but there's a big gap at my place between what's told to the fellow and what's written down. Anybody else see that? No, I'm the only one. Okay, great. But from the audience, any thoughts about how do we get faculty to write things down? Even if they're brilliant things that they've said to the fellow, when it comes to a summative or periodic evaluation, then you have a bunch of numbers. I'm going to restate that, which was I think I heard after the faculty reaches out to you kind of informally, asking them and giving them the opportunity to actually going back and editing their written feedback to reflect that and also giving them that opportunity to practice what may or may not be some uncomfortable feedback. Did I get that right? Yeah, and I'll just follow on that on an experience I had. This is how a lot of the feedback that the program director gets from the faculty is in the hallway, and when I met with the fellow at their six-monthly evaluation and gave the feedback that the faculty had told me, they were upset that the faculty didn't tell them themselves. So if you have these hallway conversations and write them down and are in the narrative evaluation, the negative is what you get and you never get the positive so then that makes the evaluation Really nice. So I like all of those thoughts towards this. I guess one other thing that I wanted to bring up was the idea of when you, if you have the hallway conversations and there's no other supporting emails and things that can kind of put you date and time, place, and then the fellow is looking at just a list of numbers, what does four out of five mean? You know, what does five out of five mean here? And how do we get more on that? Especially when your faculty are resistant to writing things down. So maybe when you give them the second chance, the third chance, you send them emails, maybe you're not getting back. What else can we do to sort of anchor that in something? And I'm interested, maybe offline or online, to hear how that's going for you. We have had a mixed review of the faculty feedback with that. But it has increased the amount of feedback that the faculty are giving verbally. It has not translated into writing it down yet, though. And just thinking of the SMART, everybody now knows, okay, SMART objectives, how to do that, and everybody can write an objective, and it's similar to the feedback. So giving them tools to say, okay, think of your competencies, what are usual words that can be used if people are doing well or not well in those competencies, or what's doing, what things you can improve, not, or giving them a framework to write down their comments on, giving them the words, giving them mnemonics to use, I think helps, it goes back to faculty development. So that's one of the first tools, really, which is if you haven't implemented a formal faculty development workshop based on narrative feedback, that's a strategy that has some real possibilities to help with this. Just because, one, it raises awareness of some of the things we've talked about, and some of the things that you all have mentioned, which is just, you may have increasing numbers of faculty at different levels of experience with medical education, they may or may not realize how important it is to actually get that same feedback that they're wanting to come to you and give you, but providing it in written form for a reason. I think that's a great thought and I think that looping back with the faculty afterwards and say, you know, hey, I gave that feedback to the fellow and I was surprised at their response. You're right, I've had that happen to me where six months later the faculty member says, oh yeah, I gave you that feedback, what happened to that feedback? So I gave it to the fellow, well nothing changed, okay. And so there's a lot of open loops, I think, and you're right, the program leadership probably has to own that to then come back in and work with the faculty and help push that forward unless you have somebody, a different structure in your department to actively engage faculty development and sort of track and trend that. Can I ask, and is that being given one-on-one back to faculty, that feedback about, sorry, that information about how their feedback was utilized or do you have like broad-based faculty outreach around that? or the trainee at that point to kind of close out the loop. Ellen and Kanta, any other points that you wanted to make about how you've been considering or how have you considered narrative feedback in your institutions and just your general experience with it? We have kind of been encouraging a lot of, we use Med Hub, finally, and we have been encouraging a lot of the sort of on-the-fly evaluations, sort of similar to what you were saying about the email. If they kind of approach us in the hallway, we ask, sort of gauge their comfort level and say, would you just put something in? And on-the-fly feedback, in this way, we have that if they forget by the end of their week or anything, something that's a little more formal than kind of keeping it in their phone and then accidentally deleting it, which has happened to a couple of our faculty members. And so we sort of just encourage, hey, if you're thinking about it, just write it down, use the Med Hub app and just write it down. Great. That starts to get us into the next question that we kind of have planned for ourselves, which was, you know, thinking through your programs, we've started talking about, you know, again, we've emphasized a lot of why we think this is important, and starting to think through some of the strategies, the concrete strategies that we've employed or we've seen employed to help promote faculty in providing narrative feedback. So, one, we've talked about various versions of faculty development, and that's really a dedicated amount of, you know, time that you incorporate into, you know, your program once a year or, you know, once every other year, especially with new faculty, to make this point and to make the points that have been brought up from the panel and from the audience around how that information will be used, why it's so important, why a numerical grading system or the checkboxes that they use in whatever, you know, program that you use aren't enough for that week-to-week feedback. And so then, there are often a couple of common barriers that we hear, and we want to talk through strategies that are incorporated with that. One of the first ones that we hear is really time. Again, sounding familiar to folks, the faculty say, I just don't have time to do all of this, to both meet with the fellow, then provide verbal feedback, then provide written feedback. It's kind of been done, you know, how do we incorporate that? And Ellen, I heard you start to talk about a strategy around taking notes. Could you provide us with a little more information on how you get all that incorporated? their next thought, and so we've sort of been encouraging either some kind of piece of paper, a little envelope in the fellow's room that they can just grab a piece of paper, write something on it, and bring it to their office, or put it in there and we'll come collect it. And then that sort of evolved into, well, if you're already writing it here, would you just consider writing it in an evaluation? And then referencing that evaluation when you actually get your real one this way, all of your thoughts are on there, and you don't have to worry about whether they're a four or a five, you can just put down I agree that time is short for everybody. And so another way to do it instead of individual thing is to maybe having whatever the group accounts that. put a framework like what the observation was and what the feedback is and everybody can log in, put their little bit, and then at the end it becomes everything comes together. Seeing a fellow for one week or a training for one week where they may be off for two days and post call for one and you don't have time to do... And just to add on to that, one of the things that we've discussed is I think there's sometimes a real mental block towards giving feedback. And I appreciate the amount of thoughtfulness and preparation that faculty want to put into it. But sometimes, sitting down and meeting with a fellow and giving some feedback is better than thinking about it for a month and then sending an email in the middle of the night. So maybe a lot like what Kantha was saying is find a way to just give feedback about a specific. But what was it about? And make sure that that gets done. Do y'all? I was just gonna ask the audience if you if you don't mind. Do y'all have a specific thing anchored in the one week two week four week process where you do like everybody gives feedback at the end of the day Friday or first thing Friday morning? Yeah, I see some head nods. Do y'all mind telling us? I love that. One of the things you just said was a cultural expectation, so change in the faculty culture so that they expect to give feedback like that. And I think one of the hard things about it and why faculty… throughout. But coaching, and it's not even the summative evaluation at the end, we're just coaches. And I think that puts the framework for faculty and trainees to receive it better. That's such a great point, you know, changing the culture and, you know, reminding everyone, even NBA players, professional athletes, they have coaches, right? We are all working on something that we are trying to improve. And so, you know, that five out of five, again, not particularly helpful for what is the thing I'm working to improve on next week. So changing the culture, it helps if it's alliterative, but, you know, whatever version of making an expectation, you know, is helpful. One of the things that one of our faculty talked to us when we were doing this internally that I really learned and have stolen is to help with that time issue is that they pull up Med Hub whenever it is, that Friday or that they're doing the feedback, and they just do it with the fellow right there. So they give them the feedback and they write it out at the same time. So essentially, at the end, they can just click submit or save, finish it later and send it off. And that helps both with the time aspect as well as the actually getting it done and, you know, delivery. So any other strategies that folks have utilized or seen done well in terms of time that we should know about to help with this kind of common barrier? It's probably the most common thing when I've talked to others about that in terms of barriers from faculty. Oh, that's great. So using a School of Medicine strategy of a faculty member who goes in and lets its feedback to them be able to provide it just to remove that barrier. That's fabulous. So talk about changing culture, changing incentives particularly institutionally has really started to help with that. Can I ask one follow-up about, do you have insight as to how that advocacy occurred to change it at the dean level? Was that kind of an organic process or was there a specific movement? Okay. Yeah. Do you get into instances where faculty... Great to work with. Submit. But even if you can't get to the narrative feedback part, at least maybe a better descriptor in terms of. And I think making it more specific so the two feedback and then why why did you say that based on what is this? So then that ties in the observation instead of okay because they did this I've been part of other learning groups where, not quite as long as like a school of medicine year, but where each learner will get a card. And so when they start with a new faculty, they actually would give them that card and say, this is my week's assignment. Right? And after a couple of weeks, they would meet with their mentor, and that card gets swapped out. So a version of, you know, your CCC conclusion might be, you know, the fellow carries around this idea of like, this is their objective for the next couple of months until they move on to a new objective. So, and so you all have started to naturally transition to not just time as a barrier, but also the quality of the feedback. So we're just going to continue with that since that's the next big barrier is not getting actual information on the evaluation. Can I ask you to clarify that a little bit more? So are you, when you say different types of feedback, you're anchoring it in a developmental milestone or you're anchoring it in a... Okay, thank you for clarifying because I think that that's So one of the things that we've struggled with is how do you take the daily tasks and operations? Oh, no, no, that was great, that was great. So those were two big barriers that we've covered. Though informally, we've covered several others along the way. So we started to talk about time as a big barrier and come up with some strategies for how to do that, whether you change it culturally, you do it by incentive. We discussed behaviors on service about getting together and actually filling it out at the same time that you're giving the fellow feedback, making space for it, finding another attending that helps facilitate that process. We started talking about the quality of the feedback and we've just heard all kinds of different ways of doing that, including what I heard as a theme there, which was just changing the expectation that what you're giving feedback after a one-week rotation isn't the entire scope of how that fellow or resident or everything is doing, right? Just focusing on one or two learning objectives, ideally that are pre-selected by the learner ahead of time, that Monday when you start, and then you know to give observation and feedback of that along the way. And so that's there. You know, emphasizing the importance of why verbal feedback needs to be written feedback. So I wanted to hear from you all at this point. Are there other barriers that you struggle with that we haven't started to talk about? And I bet as a group, we can start to troubleshoot how you might assign that. for that. I'm going to ask our panel if they have any thoughts. I would love thoughts from the room. A question of, when you have different expectations is really what it sounded like. Different expectations between how learners may be perceiving what feedback means for them and what it means for faculty to give certain types of feedback. And there was a lot in that statement, but I'm going to leave it there with saying those are the big themes that I identified. How have you all approached this idea of making sure that learners are looking to feedback as a positive thing, a thing for them to improve, as well as making sure that faculty are providing feedback that is, again, based on behaviors of clinical practice and change, as opposed to just personality traits or other things? Well, I think one of the things that we do as the program directors are we tell the fellows, hey, look, if you've gotten some feedback and you just want to vent about that feedback, just come talk to one of us and just let it all out. And then we can see if we can pick out the constructive parts of that and kind of rise from those ashes. We have something similar to something someone said with we expect our fellows to be maybe a three at the beginning of the year and then maybe go up towards a four. And setting that expectation with the fellows has been a little bit difficult, but it's kind of coming around. And so we then give them this option like, hey, we only want you to be a three. If you're a five, you're going to graduate tomorrow then. Essentially, you could be done here. You're doing my job. follow up on what you said, I think that's something that we need to let our faculty know, that if there's only one person who's giving the feedback, it can be construed as, oh, this is just, they're attacking me. But if everybody's doing it consistently and they're getting the same feedback from three different people, they're more likely to take that in and not make it a me against you type of situation. And so it's a responsibility to your colleagues as much as it's a responsibility to your learner to do that feedback effectively and have the discussions with the program director and other faculty to say, hey, I saw this, is this what you see also, or is it just my impression or my biases that are playing a role? So, and that's another good thing when you're writing it down, you should say, if any other trainee did the same thing, would I have the same feedback to just take out any personal biases you may have? So, and then there's a responsibility to society. We've all heard the stories of the trainee who gets passed along and nobody tells the next one, passed along and they graduate and then go on to do other things. So I think that feedback is a responsibility for many different, not just for the learner growth. All right, and I think that's where you get into this really difficult feedback to give. And I think, just kind of thinking about our experience, they get a lot of. Then when you get to professionalism and communication, sometimes that feedback is definitely not taken well by the trainees, but if you can couch it in, you know, hey, let's talk about. be a broader context of what professional communication and professional behavior is going to be. These are the kind of things that physicians can get in trouble with hospitals about or trouble with medical boards about. And so these are the kind of things so that you know. And so I find that when you get into those conversations, a lot of it's the fellows like, I didn't know that. And I didn't know how to, I didn't know what to do about this. I just thought Dr. So-and-so just doesn't like me, right? Or they, that I was just being, maybe kind of in your situation like where they just felt like the feedback was coming from a place of bias or coming from a different place in that. So maybe that helps. Fabulous. Thoughts? Awesome. Awesome idea. So I heard in their context building from the learner perspective, feedback. There's been lots of feedback workshops. They exist. There are more out there. We've had many here in the past. So it may be providing feedback skills for some of your faculty. And just recognizing emotion is a big part of feedback. And so helping kind of disconnect those sometimes and then finding productive ways to connect it. Just in the last couple of minutes, any other? Yeah, please. Go ahead. So this is not the most common period. But in talking with faculty who had lower condition rates in their evaluations, particularly young faculty, and asking about whether they were having fears or concerns, I think one of the things that came up is that in the context of rising expectations on faculty evaluations, evaluations of faculty teaching, there is, right or wrong, or accounted or unaccounted, concern that providing constructive feedback to learners in a non-anonymous way is going to somehow have a negative impact on them. And that they see that as a barrier and sort of want to opt out of the process. We're going to put negative vowels down. And I think it comes down to leadership and the culture. So if the trainee goes and complains about a faculty because they did this, just trying to dissect it and getting all sides in, is that truly? But there are places where the culture is, don't make the trainee upset. They're gonna complain on their ACGME survey. So just keep them happy, give them fives, give them fours. Don't upset them, don't rock the boat. And that's a culture that needs to. Yeah. Yes, please. Yeah, as a current fellow trainee, I think it's important as well to recognize from our perspective, some of this discussion about our view on feedback in terms of this specific thing, because there is also a power dynamic here as well. Absolutely. For sure, for sure. Yeah, and also, you bring up a great point that I think this spurs on more conversation with the trainees about what's the purpose of this process in graduate medical education is to make you better and equip you for future practice. And so I have to have that conversation frequently with my fellows about what are internal comments and what's externally sent. And most of this is not externally sent. I mean, if there was a consequence or there was something that's clear on paper that there was a gap in training or something like that, that's not something that's not going to be communicated. That's going to be easily seen on your CV. But internal comments, I think maybe we just need to put that in context. So we need to do better to talk about why this was said and how we tie that to your training and how that is advancing. And if there's remediation, does this remediation have a start and an end to it? Was this remediation successfully completed and now we're done? Or is this, there's an escalation in this process and so this is a letter to the file or this is whatnot? So maybe we need to do some more explanation. And I think you're right. I think you can make it a bi-directional feedback. Let me help you be a better clinician and you help me be a better educator. And that opens up channels of communication, I think. That's great. And with that, I think we start, one of our first comments was about a version of this, actually of faculty getting feedback about their faculty. So I'm going to use that time to actually close us off here because we could do this for a lot. As you all have really highlighted that it's not one simple thing, right? It's really, there are a lot of barriers. They're very individualized to your institution. So it does require some time thinking about this at your institution. But hopefully we've highlighted some of the common barriers that many institutions have seen some of the common ideas of how people have started to think about addressing some of those barriers that hopefully you can utilize at your own institutions for the version of those challenges that you have. And we're certainly happy to continue kind of talking about this offline. So thank you so much for your attention. Thank you so much to the panel. I really appreciate it. Thank you.

narrative feedback

educational programs

improving feedback quality

faculty development workshops

specific and actionable feedback

changing feedback culture

anchoring feedback

learner growth