Program Evaluation

Narrative Learning Program Evaluation

Group Member	Role	Commented On
Laticia Alexander	Reflection; Table	Group 1 and 3
Julie Furnish	Student Response; Reflection	Group 1 and 5
Nichole Mann	Student Response; Reflection	Groups 3 & 4
Carrie Reisner	Evaluation; Reflection; Review; Edit	Groups 1 & 3
Pam Shawl	Evaluation; Reflection; Review; Edit	Group 1 and 4

Professional Evaluations

Prudence Nelson’s Evaluation

    Prudence Nelson has been a First-Year Seminar instructor at Indiana University East for the past eight years.  She teaches the course full-time in the fall semester (five sections) and part-time in the spring semester (two) sections.  Prudence uses a variety of teaching techniques, including the use of narrative, in order to help students develop the skills needed to transition to college.  One of her most popular activities is to have her classes follow the CNN Hero series and have students discuss what makes each of the individual stories special, then vote on which one should win.  Prior to teaching FYS, Prudence taught first-year English composition courses and worked in the former tutoring center at IU East.

Prudence provided a well thought out review of the program. It was very positive overall and spoke to the purpose of using narrative in this particular learning context.  She noted the value of using narrative with adult students who are returning to college, potentially after a significant absence from education, and how that can benefit students who may not necessarily connect informal learning experiences with the formal learning they are about to embark on.  In regards to the cross-cultural interviews, she noted the importance of situating learners in a cultural context and how that helps them to learn more deeply about themselves.

    In addition to positive comments, Prudence also provided some valuable suggestions for improving the program.  These included providing more specific prompts for writing the final reflection, considering how students will share their narratives with other students, and how the narratives will be assessed by the course instructor.  These are all excellent suggestions that will help to improve the overall quality of this narrative program.  

Student Response

          Ms. Nelson’s evaluation of our program was encouraging. She seemed to have a clear understanding of our program design, and it was positive to see that, as an expert in adult learning, she felt it would be effective for increasing non-traditional learners’ understanding and use of personal narrative as a learning tool.

Ms. Nelson made some key suggestions that would contribute greatly to our program design regarding clarification of activity objectives. She expressed that the wording of our questions on how the interviews impacted their learning for the reflective activities may not be focused enough to tie them directly to our learning objectives. I agree with this assessment. Being very focused with how we choose to ask our questions will give students a more concrete idea of what they are supposed to do and help them understand the purpose of the activity more thoroughly. She did suggest, however, broadening the initial question of the program (“What qualities and experiences make you the student you have become, and how does your role as a student fit into your sense of identity?”). This could be pursued either way. Her rationale that non-traditional students may have a hard time viewing themselves as students because it’s been some time since they were in formal educational settings, is a useful perspective, but it could also be argued that by focusing in on their role as a student, the program encourages them to consider how they will integrate their previous, non-educational experiences into the educational setting.

   The other especially valuable contribution Ms. Nelson provided was her two “areas of curiosity.” These were two questions not addressed by the program design, but critical to its success:

• Are the interviews students complete shared with other students? Do students respond to other students’ narratives and interviews on the discussion board? If so, what is the criteria for their responses?
• How are the narratives assessed? What is the instructor’s role in evaluating the narratives and interviews?

 For the first question, it is important that it be established early how the interviews and reflections will be used. This may impact both the students’ and the interviewees’ willingness to be candid. Sharing the interviews compounds everyone’s growth in the class, but keeping them private creates a more open space. Our solution is to split the middle-the students will discuss their interviews and engage with others about what they learned within the class, but their submitted reflections about their own experiences, thoughts, and feelings will be privately shared with the instructor.

 Assessment is an important component of any educational program, and it was a definite oversight not to address it more thoroughly in our design. It would be counterproductive to suggest that one’s personal narrative could be wrong. By definition, it is the experience of the learner as he or she perceives it. Assessment would have to be based on technical proficiency (did the student adhere to a rubric of required tasks? Is the assignment of appropriate length and structure?) and “good faith” work toward the objectives of the assignment. A student cannot be forced to have a breakthrough, and transformational experiences can hardly be reliably measured by an instructor even if it could be forced. However, we can assess a student’s willingness to engage by the way a student describes interviews, the questions they ask, and their choices they make about how to approach the assignment. The assessment measures can be communicated to students in a way that both clarifies expectations and gives a more complete picture of what the student should get out of the assignments; for example, an assessment measure may be, “Student includes 1-3 interview questions that specifically request a story or opinion regarding a social issue likely to be relevant to the interviewee.”

 Ms. Nelson’s opinions were extremely valuable, and highlighted specific areas of interest to improve our program design and better achieve our objectives with this program.

Elaine Bruns’ Evaluation

Elaine Bruns is Adult Education administrator at Fayette County Schools for Indiana Workforce Region 6. Elaine works with adults who are pursuing their high school equivalency diploma or refreshing their skills to prepare for the workforce, training or personal goals. Elaine is also a former adjunct instructor teaching first year college students in first-year English courses.  

As a result of Elaine’s experience, she understands and appreciates the impact a FYS course has on a new college student. She also feels that a required FYS course can help with retention and student success in that the FYS course can help the student to feel connected to the institution. Elaine provided a positive review of our program and stated, “Using narrative learning seems an excellent approach as it encourages deep introspection which may be new and uncomfortable, yet allows the students to reflect and share using storytelling which feels familiar.”

Elaine noted that incorporating narrative learning in a non-traditional adult FYS course made sense, because with people like to tell stories whether it be in person or through social media. She felt that narrative learning would encourage the students to share and learn from each other. Elaine especially felt that narrative learning in a non-traditional FYS course would broaden the students’ appreciation of diversity.

Elaine also made suggestions for improving the program. She suggested stricter guidelines for assignments such as adding research components to the course so the class would feel more collegiate. She also felt that appreciation of diversity could be incorporated by having students interview three people asking the same set of questions of each person: one of a different race, one of a different gender and one of a different sexual orientation.

Lastly, Elaine brought up a very valid concern about providing our FYS course as an online course and steering conversation. Elaine discussed how when incorporating narrative learning in the classroom that this encourages students to talk. In the classroom environment, the teacher can steer the conversation more easily when the conversation gets too far off topic.  However, in the online classroom, the teacher is not always in the conversation at that time in order to steer the conversation. In this case, Elaine suggested the instructor decide whether posts will be edited or deleted.

Student Response

    Ms. Bruns had a lot of very good support and suggestions to improve the FYS in a lot of positive ways. Like Prudence Nelson, she quickly noted that she approved of the narrative learning style and that it is a great one to use for people in general. She notes that this style of learning is useful due to its introspective properties about new and uncomfortable environments, but allows the learner to reflect their own experiences as well.

    One of the first concerns that Elaine had was students feeling suppressed due another’s privileged position, especially if this individual speaks too often or first. She suggested using Peggy McIntosh’s “White Privilege: Unpacking the Invisible Knapsack” as a good way to introduce this concept, but expand it to all forms of privilege, including economic class, social sphere, gender, religion, etc.

    I felt that this was a good point to bring up, and to help students understand and allow an open environment to share stories and experiences based on our different weekly discussions and open narratives. Since it is human nature to talk about ourselves, it will make everyone feel more inclusive to include this essay and discussions on it for everyone to better understand. It is one read that should be required at the beginning of the program though.

    The next concern outlined by Elaine was due to human's innate desire to discuss themselves, where do we draw the line as to what is an acceptable story or discussion? She mentioned witnessing an online classroom discussion go from a story of a recovering addict to a current list of drug dealers and their addresses. In an online classroom setting it is difficult to steer students toward a more positive and productive direction. This is where she suggests we have guidelines as to where the instructor deletes or edits the post shown.

The thought of this happening, did not honestly even occur to me happening, but I could see this being an issue. Especially in an environment where we are encouraging narrative discussions that can easily get out of hand. If the previously mentioned essay would be an included read at the beginning of the class, it may help cut down on some of the negative comments that could stem from discussions. Outside of that it would be up to the instructor to go through and delete or edit out any discussions that have other individuals personal information, or derogatory statements. From there the instructor would likely have to speak to the student making these comments, and potentially report it to the school.

The last suggestion that Ms. Burns had for our project was to try and make the guidelines for the interviews and discussion boards more clear cut. Some of her suggestions were to have students include a historical/ fictional character they most resemble and why in their personal narrative. This provides a more collegiate feel to the project, since they would have to do some research on this, but still have an open feeling to it. One of her other suggestions was that for the interviews. She suggested to have the students pick another individual that is of a different race,  gender, and sexual orientation. This is a concept we had actually discussed while laying out the groundwork for our project and decided against it. As a group we felt that this could become a bit of an issue. It was felt that the race and religion (though Elaine did not specifically mention religion) could be an issue with profiling and separation of religion and school. This is why we designed it to have students pick three different people, since we all come from different cultures and backgrounds.

In Elaine’s final suggestion she did state that we should require a few different questions, which I think is a good idea. This would help with the more guideline feel that is associated with school. Most of her suggestions were fun questions, but ones that would provide insight to the individual being interviewed. A few of them included:

• What is your biggest fear?
• Do you think you have ever been denied an opportunity at work or home because of your race/ gender/ orientation/ etc.?
• Do you feel comfortable kissing your significant other in public? Why/ why not?

    Elaine’s review was exceedingly helpful to our program and pinpointed some specific areas that could be improved. Her evaluations are ones that would better enhance our program and achieve the intended goals of this classroom program.

Group Reflection

    Our group worked especially well together over the course of this project.  From the initial idea for the program, to the identification of an audience, to the program design and evaluation, the group communicated quickly and effectively through email and utilized Google Docs in order to be able to share in the writing process.  Everyone provided feedback along the way and each individual brought strengths to the group.  

    One of the more challenging aspects of the evaluation part of the project was the reliance on outside professionals to get the evaluations completed.  Both of the evaluators for our project are busy people and their time is very limited.  Prudence and Elaine were willing to help out but each had other obligations to juggle - Prudence was preparing to travel out of state and Elaine was preparing to submit a grant.  We appreciate that both were able to work quickly to complete the evaluations within our timeframe.  Once each of the evaluations were received they were sent to the group for review.  We then proceeded to write the student responses and put the table together.

It is common in the planning phases of any new project to fear that a fatal flaw will send you “back to the drawing board,” so it was encouraging that the program reviewers were so positive about the project. We incorporated a wide range of factors into our project, and the program evaluations brought to light some of the fundamental pieces that were missing. It was a pleasant surprise that the reviewers were able to understand all of the essential goals we were attempting to address and build on those things so effectively. We started with a broad project, and the evaluation phase brought out some great ideas for targeting and assessing it well.

Table

Evaluator	Ideas for Improving Program	Revisions/Responses
Elaine Bruns	Using Peggy McIntosh’s “White Privilege: Unpacking the Invisible Knapsack”	Absolutely a great suggestion and a great way to start the semester.  The exercise is a great way to make an online classroom feel like a safe place to have discussion.
Elaine Bruns	Using Guideline Questions such as: What is your biggest fear? Do you think you have ever been denied an opportunity at work or home because of your race/ gender/ orientation/ etc.? Do you feel comfortable kissing your significant other in public? Why/ why not? What  historical/ fictional character you most resemble and why	We felt that these guideline questions would be a fun way to start a narrative.  It would also greatly assist in making the online environment a safe place to openly discuss.
Prudence Nelson	Establish early how the interviews and reflections will be used.  Will they be shared on the discussion board or just with the professor?	Keeping the interviews private provides a more open space but sharing them impacts everyone’s growth.  We decided to split the assignment.  The interviews will be posted on the discussion board and the reflections will be submitted to the Professor directly.
Prudence Nelson	How are we going to assess the student’s personal narratives?  And interviews?	We felt that it would be impossible to assess based on content on the personal narrative.  We then decided to create a rubric that is based on technical proficiency.  Is it the appropriate length?  Is it the appropriate structure?  Was there good faith in meeting the assignments objectives? For the interviews our rubric would address these questions.  Did the interview questions invoke 1-3 stories or in depth opinions on a social issue?  Did the interviewer present the interview in a unique way?  What technology was used to conduct the interview?

Appendix - A

https://docs.google.com/document/d/1YFW7-qJYom_d54lIYem1jLHYDidZW4jArR7ponm3RBE/edit?usp=sharing

Narrative Learning Program Design

Evaluation by Prudence Nelson (Indiana University East) - full time First Year Seminar instructor in the fall and adjunct FYS instructor in the spring 

My immediate impression of the narrative learning program is how suitable and beneficial it would be for non-traditional adult learners, whether utilized for online First Year Seminar courses or within a classroom. In meaningful ways, the narrative requirements draw the individual dimensions of students’ experiences into dialog with the work they are beginning and the future the student is imagining and pursuing and suggests a sense of the opportunities for self-efficacy and agency that lie ahead. The narrative learning program also offers a conception of learning that is more inviting than the prevailing sense non-traditional students may bring to college and has strong potential for engaging students and enabling their participation in a group. 

The use of personal narratives also enable non-traditional students, through personal discovery, to identify their own learning needs and strengths and at the same time helps them articulate and define their learning/education purpose and process. The cross-cultural interviews draw on broader narratives circulating in communities and society (such as race, class, gender, ethnicity, masculinity and femininity, and culture), positioning the adult learner in relation to diversity, which signifies difference as something to be sought in learning and that diversity is enriching. This doesn’t need equal weight with the focus on the students’ personal narrative, but it is important to acknowledge not simply as an element of cross-cultural learning, but as fundamental in understanding the self.

While the narrative learning program’s primary theme focuses on each student’s educational journey, the initial question: “What qualities and experiences make you the student you have become, and how does your role as a student fit into your sense of identity?” is narrow. The program is designed for adults (who are often in transition) returning to college; perhaps, their “formal” education and memories of being a student are in the distant past. The guidelines for the personal narrative might “make room” for reflection on what they have learned from more recent experiences (such as being in the military, raising children, employment, having an illness or being a care giving) and how the experiences shaped their sense of identity and competence in begin college. This theme could be reinforced in the cross-cultural-interview assignments as well. 

The final reflective revision provides students with a pivotal opportunity for personal introspection and concluding critical reflections of their FYS narrative journey. However, I find the instructions for integrating the interview material with students’ educational biography somewhat vague (“It should also incorporate whether they learned anything new from the interviews”) and broad (“students explore what their own stories and those of others teach them about the world”) and could be more clearly defined.

Additional exploration and consideration directed at how to establish, in an online learning environment, a supportive, nonthreatening, and open atmosphere for sharing the narratives would help alleviate issues about how comfortable and safe students feel presenting/revealing themselves to other students as well as to the instructor.

Although not included in the expectations of the program design, two areas of curiosity include:

• Are the interviews students complete shared with other students? Do students respond to other students’ narratives and interviews on the discussion board? If so, what is the criteria for their responses?

• How are the narratives assessed? What is the instructor’s role in evaluating the narratives and interviews?

One other question I would raise is whether there is a tension between the FYS as a site of engagement and narrative learning/reflection, and as a site of compliance with Federal regulations and university policies (for example, Title IX), and their growing role in the curriculum. 

These aren’t criticisms so much as reflections on what the proposed seminar direction might elude and what it might also address. I am impressed by what the narrative learning group has proposed, and with the potential of narrative learning as a way of situating and supporting non-traditional college students in the FYS online course. The rationale is particularly persuasive. 

 Appendix - B


https://docs.google.com/document/d/1SFyxH1UftKO89mJJXsXJesFn1K6uzA1QhE2ugXDVJbw/edit?usp=sharing

Narrative Learning Group Project

Reflection completed by Elaine Bruns: current Adult Education administrator, equipping students with resources necessary to achieve the high school equivalency diploma and complete college and career goals by entering the workforce or a postsecondary institution.
            As a former adjunct instructor and collegiate student services advocate, I understand and appreciate the impact a FYS course has on the new college student. A required FYS course for the incoming non-traditional student is even more crucial, sometimes defining the success and retention of that returning student. It is vital, then, that the FYS course be designed to connect the students to each other and the institution.  Using narrative learning seems an excellent approach as it encourages deep introspection which may be new and uncomfortable,  yet allows the students to reflect and share using story-telling which feels familiar.
            First and foremost, from my experiences, people love to talk about themselves (see the rise of social networking, facebook, instagram, etc.). This is not necessarily a narcissistic flaw, rather I would argue, a survival strategy.  In talking about my own experiences, I encourage those around to share with me. When I was pulled over and had a positive interaction with the police officer, which resulted in a warning only, I shared. When my 2-year-old son fell in the bathtub, requiring stitches in his chin, I shared. When I book an excellent travel deal to Las Vegas and am proud of my Expedia sleuthing, I share.  I don't share to exploit all my failures or to gloat in my wins. I share because something I have done may encourage learning. I expect my peers to do the same, so that I may learn about the world around me and in turn, exist more comfortable, more aware, more intelligent. Narrative learning in a non-traditional FYS course embraces this story-telling-as-survival ideology. It encourages students to share so that they may grow and succeed.
            My own personal philosophy embraces narrative learning's most outstanding feature for an FYS course: appreciation of diversity.  The different stories adults bring with them into a classroom are absolutely astonishing. Some of the stories are harrowing tales of disaster, others painfully normal, but none of them in a non-traditional collegiate setting are 'perfect'.  These imperfect stories help students connect with each other and encourage a sense of pride in their own paths.  Narrative learning will help broaden the understanding of others' backgrounds, almost forcing empathy for situations one student may have never envisioned. 
            In my own adult education classrooms, the stories told can become stories suppressed if those in more privileged positions speak first and/or too often. Addressing systems of privilege should occur day one (a recommended tool to include is Peggy McIntosh's "White Privilege: Unpacking the Invisible Knapsack") so that students can start to accept the oftentimes foreign idea.  Note that while the particular tool recommended has white privilege in its title, it can and should be expanded to include the privileges that come when being in the 'right' class, economic bracket, social sphere, gender, sexuality, religion, etc. While it is a tough pill to swallow for some (straight, white, middle-class, Christian men), it can assist in conversations going forward, allowing the instructor and/or students to point out when someone may be speaking from a position of privilege.
            Sometimes, narrative learning in non-traditional adult classrooms encourages stories shared to become stories with way too much information about way too many people with way too little tact.  I think this is because the students are eager to share with those who want to listen; a notion many have never felt wholeheartedly. In my classrooms, if someone perceives another of having a tough road to hoe, the conversation can quickly become a dramatic outpouring of descriptive proof regarding tough roads and the difficulty in hoeing.  I've witnessed classroom conversation take a turn from being about the struggles as a recovering addict to a 'hit list' of current drug dealers and where they live. While there are still benefits to this conversation, the instructor must be equipped with the grace it will take to move students toward more productive discussion.  In an online setting, it should be decided whether any posts would ever be deleted or edited.
            To avoid the FYS course becoming  "that class you have to take for no reason", I might encourage stricter guidelines in assignments. It is already a very open, discussion-based, 'there is no wrong answer' course, so further allowing students to just write about themselves, and choose any three people for the interview may be a little too lax. Asking the students to incorporate ways they are each privileged into their individual beginning narratives then again at the end could help determine what level of understanding the student has in regards to privilege. Perhaps in their individual narratives, each student picks a historical or fictional character they most resemble and explain why (I relate to Frankie Heck from The Middle, and unfortunately could provide references), or each student chooses an historical or fictional character they would like to be and why.  This could just add a bit more research/collegiate-feel to the assignment.  In choosing interviewees, maybe they instead must choose one who is a different race, one who is a different gender and one who is a different sexual orientation. They must ask each at least the same five questions:

1. What do you fear when you're pulled over by a police offer?
2. Do you feel comfortable kissing your significant other in public? Why/why not?
3. Do you think you've ever been denied an opportunity at work or home because of your race/gender/orientation/etc.?
4. Do you think you've ever been given and opportunity because of your race/gender/orientation/etc.?
5. What is your biggest fear?