IMLS
-- Assessing Student Learning Outcomes: Training Academic Librarians
Assessing Information Literacy
in Education 110 (Introduction to Education) at Highline Community College
Institution
Description
Highline Community College is a public, suburban
community college about 15 miles south of Seattle, Washington. We have a diverse student body both in terms of ethnicity and
academic pursuit. In a typical
quarter, we serve approximately 9,000 students (5,000 FTE) taking Academic
Transfer (44%), Vocational (23%), Basic skills (22%), and Developmental (10%)
classes. Highline’s library has
an active and growing program of information literacy instruction, supported by
the college administration. In
the past decade, our staff of full-time faculty librarians has quadrupled in
size (from 1 to 4) and the numbers of classes we regularly teach has tripled.
Highline also has a history of encouraging collaboration, including
faculty/librarian collaboration. Librarians
work closely with faculty to design instruction tailored to course outcomes and
assignments, and faculty in the disciplines are becoming accustomed to our
asking questions about their learning outcomes at the time they schedule
instruction. Highline is also
committed to student learning assessment. We
have an assessment committee, with representatives from each division plus the
library, and each academic department (including the library) has published
outcomes. One of the functions of
this committee is to promote assessment activities at the course and department
levels, supported in part with an annual influx of state funds.
The college is currently involved in the accreditation process, and has
included significant discussion of the library’s information literacy program
in both Standard 2 (Instructional Program) and Standard 5 (Library) of its
self-study. In this environment, it
was actually quite easy to find partners and support for an information literacy
assessment project.
Project Participants
The participants in this project were myself, Dana
Franks, Lead Instruction Librarian, and Joan Graham, a member of our Education
faculty. We worked with Education
110 – Introduction to Education. The
class focus is on the major educational philosophies
and theories, historical events, and curriculum models that influence American
education, with a practical focus on how these
are manifested locally in Washington State.
The class includes a research component, with students making an
end-or-quarter group presentation exploring a significant education issue of
their own choosing.
Project Description
The Collaborative Process: I met with my
partner, Education instructor Joan Graham, over the summer, 2001.
We began by examining course outcomes and assignments for Education 110.
After doing so, we decided to focus on providing students with the
information literacy skills they would need to complete the course’s major
assignment: an end-of-quarter group presentation, accompanied by a brief
summarizing essay and bibliography. To
identify the necessary skills, I gave Joan a brief version of our information
literacy outcomes (which are based on the ACRL standards) and asked her to
identify those she thought the students would need in order to complete this
assignment. There was fruitful
discussion surrounding this activity during which I learned more about her
discipline, her class, and past student performance on the assignment.
Joan had several questions too. Specifically,
she needed clarification of some of the outcomes and wanted to know who I
thought was responsible for the instruction and assessment.
My answer to the latter was that it was up to us to decide that, within
the context of this course, but I supposed we would each have a share of that
responsibility. Happily, we agreed
on that point. In the end, we were
also in complete agreement on the selection of information literacy outcomes to
address.
Developing Objectives and Assessment Criteria: Despite the fact that we had not yet developed specific learning objectives, Joan and I decided to examine potential assessment instruments next. With our selected outcomes in hand, we examined the assignments already used in Education 110 to determine the extent to which they could serve our needs. We found that there were two assignments that would work well for several of the outcomes. I developed a third instrument, (an annotated bibliography). We believed that, taken together, these three instruments would address all the outcomes.
With the outcomes
and the assignments laid out in front of me, I completed a “Planning for
Assessment” table with the following columns:
|
information literacy standard |
performance indicator |
outcome |
objective |
instrument and method |
criteria |
|
(3) |
(2) |
(1) |
(4) |
(5) |
(6) |
I began by inserting the outcomes we had selected (1), listing them according to the order in which they appear in the ACRL Standards. I then filled in the columns to the left – the corresponding Performance Indicators (2) and Information Literacy Standards (3), again from the ACRL Standards. Using these and the assignments as guides, I then wrote specific student learning Objectives (4). It was relatively simple then to plug in the assignment name and section that would be used to assess that objective. These were our Instruments and Methods (5). When it appeared that the assignment was inadequate, we modified it to meet the need. Finally, I wrote Criteria for assessing student performance on that piece of the assignment/instrument (6). This task presented us with the dilemma of ascertaining the degree of competence we expected of our students.
Degrees of Competence: Education 110 is an
introductory course offered fall quarter, so it attracts a number of students
who are new to the college. For
that reason, we decided that we should approach the students as novices to the
information literacy field and assess these skills at a basic level.
Consider, for instance, an advanced or upper division student applying
“critical thinking and problem solving skills to analyze and evaluate
information,” (one of our outcomes). That
student might be asked to do more extensive research, or apply previously
acquired subject expertise in evaluating a source.
A novice student, on the other hand, might simply be asked to identify
clues to the reliability of a source, or examine, and communicate their own
thinking process about evaluating it. It
is in this way that Joan and I decided that we would teach to, and assess for,
an introductory level of information literacy competency.
Developing the
Rubrics: Given the assessment
tools we were using, we determined that grading rubrics would make the
assessment phase of our project easier and more reliable.
To develop the rubrics, I went back to the “Planning for Assessment”
table and clipped the “Objective” and “Criteria” columns.
The objective became what I was assessing.
The criteria became the highest level performance criteria (see below).
What
was left was to define lower levels of competence.
I experimented with various numbers of columns (up to five) and various
terms to describe the levels of competence, but found that I was splitting hairs
when I had more than three. The
rubric that emerged had the following columns:
|
objective |
criteria |
|
|
|
objective |
exemplary |
competent |
emergent |
|
|
|
|
|
Analysis:
When grading, I attach a copy of the grading rubric to each student’s
assignment. I make my comments
right on the rubric with references to their work. The rubric saves my having to
repeat comments about common errors from one paper to the next, leaving me more
time to add comments tailored to their performance.
The more specific the criteria are, the more successful this will be both
as a reliable assessment tool and as a communication device for the student.
Challenges
and Conclusion: We faced various
challenges in this process. For
instance, writing the criteria without examples of student work was tricky
because the degree of specificity we sought was difficult to imagine.
As student work did start to come in, we found that their performance
wasn’t as strong as we had hoped it would be.
Ideally, the rubrics would have been distributed to the students at the
time they received their assignments. We
just hadn’t developed them in time to do that this year.
Another approach we may consider is to add a self-assessment step to the
process, requiring students to apply the rubric to their own work before turning
it in. In the end, we decided to
use this year as a time to experiment with, and fine-tune, all the materials we
developed – the outcomes, the objectives, the assignments and the rubrics –
and apply these new strategies in 2002-03.
Assessment Instruments
Two of our assessment instruments were assignments,
with corresponding rubrics. The
other instrument was the group presentation itself for which Joan had designed a
grading rubric based on the education objectives. Even though there was overlap between the education and
information literacy objectives for this assignment, they didn’t match up the
way we would have liked. So, I
created a rubric that piggybacked on the first, for assessing the information
literacy objectives. As it turned
out, this was unworkable because the first rubric didn’t provide enough
information to reliably complete the second.
Furthermore, the two are redundant and confusing to the students.
My goal before fall quarter 2002, then, is to work with Joan to integrate
these instruments. (For instruments, see pages Franks-7 to Franks-17.)
Results
The Data: The students’ achievement in this class was somewhat disappointing. Information literacy instruction had been provided to this class in previous years, and though it had not been assessed, we were trying to address perceived shortcomings. We made several changes in how we worked with the students, informing them of the information literacy expectations and modifying both assignments and pedagogy. Based on our first quarter’s results, though, we don’t appear to have made a significant difference in student performance. With the data in hand, though, we’ll be able to track our progress from here on out.
Following are the objectives for which the majority of the students scored at an “Exemplary” level:
Students will select one each: a background source, a vocabulary source, a statistical source and a current events source related to their topic
Following are the objectives for which half of the students scored at an “Exemplary” level:
Following are the objectives for which no students scored at an “Exemplary” level:
Following
are the objectives for which the majority of the students scored
at a “Competent” level:
Following are the objectives for which half
of the students scored at a “Competent” level:
Following are the objectives for which the
majority of the students scored at an “Emergent” level:
Following are the objectives for which half
of the students scored at an “Emergent” level:
Students will meet with group members to refine topic (Section 1)
Students will be able to describe how each of their four sources provided the information they needed
Response: It’s tempting to fall into the mindset that the problem rests with the students: “I taught it to them yesterday – they just didn’t learn it!” Of course, that won’t work. So instead, we need to look at what changes we need to make to improve student success. One way of looking at assessment – what it is and why we do it – was articulated beautifully by Debra Gilchrist at a presentation to the Immersion class of 2000. The purpose of assessment, she says, is:
That last bullet is now the focus of our efforts. It is clear from our results that changes need to be made in several areas if we are to improve student learning. Some approaches we will examine are:
Assessment Instruments
Preliminary
preparation for the
Topic:
__________________________________________________________________
Names and duties of group members:
|
1) |
Names _____________________________ _____________________________ _____________________________ _____________________________ |
Specific duties _____________________________ _____________________________ _____________________________ _____________________________ |
Why is this topic important?
_____________________________________________________________________
_____________________________________________________________________
Put your topic into a question:
_____________________________________________________________________
_____________________________________________________________________
Complete the “Great Finds” assignment
List five main concepts that you are finding out (state in complete sentence):
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
What are your tentative conclusions as a result of your research thus far?
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
List two or three methods you will use to present your information:
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
·
This form is to be submitted by <date> ·
Rubric for Scoring: Preliminary
Preparation Sheet
Education
110
|
Criteria for Evaluation |
3 – Exemplary |
2 – Competent |
1 – Emergent |
|
Students will meet with group members to refine topic
(Section 1) |
Group members duties are described on the Preliminary
Preparation sheet in a way that is reflects group process and planning |
Group members names are listed and a brief description is
given of their research responsibilities |
Students names may or may not be listed but specific duties
are not addressed |
|
Students will clearly articulate a focused research topic
(Section 3) |
Topic is stated in terms of a focused question that
identifies a claim, or argument to explore and clearly suggests what
information is needed |
Topic is stated in terms of a question which is a complete
sentence but which does not identify a claim or argument to explore or
does not clearly suggest what information is needed |
Topic is not in the form of a question or is described in a
short phrase or string of keywords or is not described at all. |
|
Students will identify 5 concepts, implicit in the main
topic, for further investigation |
The concepts are implicit in the main topic, demonstrate a
clear understanding of its depth and breadth, and clearly suggest what
information is needed |
At least three concepts are listed and they are related to
the main topic they do not develop the depth and breadth of the topic |
Two or fewer concepts are listed or more are listed but the
don’t develop the main topic beyond what is stated in the main topic
question |
|
Student can draw tentative conclusions about the topic
based on preliminary research |
Tentative conclusion statement reflects comprehension and
synthesis of the previously identified main concepts |
Tentative conclusion summarizes one or two sources but does
not synthesize the information |
Tentative conclusion is reflective of only one source or is
unsubstantiated |
|
Student can express these conclusions in a clear statement |
Tentative conclusion statement makes a claim and is a
complete thought, clearly expressed |
Tentative conclusion summarizes one or more points but does
not make a claim, or, it makes a claim but does not communicate it clearly |
Tentative conclusion is inconclusive or expressed in short
phrases or is missing altogether |
Great Finds
You are doing research to find information for your
group presentations. Now you get to
show us how well you can do this. Specifically,
we’d like to see that you can:
Your
Assignment:
As a group you will assemble an annotated list of 4 "Great Finds". You may work individually or combine resources, but all members must contribute. This list should contain one of each of the following:
Cite each of your sources correctly using APA format (refer
to reference sheet or online guide: http://flightline.highline.ctc.edu/reference/Userguides/ug_apastyle.htm.
For each one, write a brief annotation stating:
Criteria for assessment :
Rubric
for Scoring: Great Finds Assignment
Education
110
|
Criteria for Evaluation |
3 – Exemplary |
2 – Competent |
1 – Emergent |
|
Students will identify at least one source that
substantially increases their vocabulary
surrounding their topics |
The selected vocabulary source and its annotation clearly
demonstrate a substantial increase in vocabulary useful for further
research on the topic |
The selected vocabulary source provides some terminology
pertinent to the topic, but the annotation does not demonstrate that it
substantially increases vocabulary useful for further research on the
topic |
The annotation does not indicate how the source provided
relevant vocabulary useful for subsequent searches. |
|
Students will select one each: a background source, a
vocabulary source, a statistical source and a current events source
related to their topic |
All four types of sources are listed and they fall under
the four assigned categories |
Two or three sources are listed and they fall under the
assigned categories, or four sources are listed and they don’t all fall
under the assigned categories. |
One or fewer sources are listed, or, more are listed but
they don’t fall under the assigned categories |
|
Students will identify at least four library resources, in
at least two formats, that, taken together, provide substantial treatment
of their topic |
Resources are in at least two formats and, taken together,
they provide substantial treatment of the topic and concepts articulated
on the Preliminary Preparation sheet, number 5. |
Resources are all in one format or they provide essentially
the same information as one another though they do treat the main topic |
There are fewer than four resources or they are only
marginally relevant to the topic |
|
Students will be able to describe how each of their four
sources provided the information they needed |
Annotations clearly state how the sources provided the
required information |
Annotations describe the sources but don’t sufficiently
relate them to the assigned categories |
Assignment is not annotated or annotations are limited to
short, vague phrases |
|
Students will be able to describe the process they used to
evaluate the reliability, validity, accuracy, authority, timeliness, and
point of view or bias for each of these sources |
Students clearly describe the evaluative process they used
in judging the reliability, validity, accuracy, authority, timeliness, and
point of view or bias for each of their four sources |
Students give opinions about the reliability, validity,
accuracy, authority, timeliness, and point of view or bias for each of
their four sources but don’t describe the evaluative process they used |
Students do not evaluate the sources |
|
Students will cite four sources using APA citation style |
Sources are cited correctly using APA citation style |
For each citation, three or more of the following - author,
title, journal title, volume, date, pages or URL - are provided, but they
are not correctly formatted |
Fewer than three elements of any citation are provided and
they are not correctly formatted. |
Intro to Education Name ________________________
Evaluation
for Group Presentation
The following elements will be considered in
evaluating the group presentations. You
will complete Section 4 (required) and the Comments (opt.) and submit
this form on the day of your presentation.
Criteria will be rated according to the scale of
5 =“WOW”
4=solid, substantial 3=met
expectations 2=needs help 1=poorly done
|
1. Elements of presentation |
Scored
by instructor |
||||
|
· Quality of presentation: clear and audible speech, interesting minimal reliance on notes, eye contact, attitude, appropriate attire |
5 |
4 |
3 |
2 |
1 |
|
·
Content/knowledge of
subject:
thorough presentation of information, obvious research done, synthesis
and application of information |
5 |
4 |
3 |
2 |
1 |
|
· Organization/structure: introduction, main points presented, “flow” of presentation, closure, keeping within allotted time |
5 |
4 |
3 |
2 |
1 |
|
· Presentation/style: uniqueness, creativity |
5 |
4 |
3 |
2 |
1 |
|
· Fair/factual: balanced presentation of views and comparisons, use of reliable facts and statistics |
5 |
4 |
3 |
2 |
1 |
|
2. Paper |
|
|
|
|
|
|
·
Overview of presentation: coverage of main points |
5 |
4 |
3 |
2 |
1 |
|
·
Organization: citing of sources, SPG |
5 |
4 |
3 |
2 |
1 |
|
3. Hand-out |
|
|
|
|
|
|
·
Content: information relevant to and representative
of presentation |
5 |
4 |
3 |
2 |
1 |
|
·
Organization: format, professional appearance, SPG |
5 |
4 |
3 |
2 |
1 |
Each student will complete this portion for each
member of the group (+ self)
|
4. Peer evaluation |
|
|
|
|
|
|
· See rubric below to determine the peer scores you will give for this section. Final peer grade for each student will be determined from group score averages. |
|||||
|
Name (self)
___________________________________________ |
5 |
4 |
3 |
2 |
1 |
|
Name _______________________________________________ |
5 |
4 |
3 |
2 |
1 |
|
Name _______________________________________________ |
5 |
4 |
3 |
2 |
1 |
|
Name _______________________________________________ |
5 |
4 |
3 |
2 |
1 |
Criteria for peer group grading:
|
5 points |
4 points |
3 points |
2 points |
1 point |
|
Attended all workdays with a “wow” level of · Significant contributions · Cooperation · Skills in com-promise and negotiation · Shared responsibility · Resourcefulness · enthusiasm |
Missed no more than one workday plus a high level of the items in the first column |
Missed no more than two workdays plus a moderate amount and level of the items in the first column |
Missed two work- days and/or con-tributed at a low level of the items in the first column and/or/negatively impacted the group |
Significant absence on workdays and/or lack of participation and/or negatively impacted the group. |
Rubric
for Scoring: Evaluation for Group Presentation
Education 110
|
Criteria
for Evaluation |
3
– Exemplary |
2
– Competent |
1
– Emergent |
|
Students will meet with group members to refine topic
(section 4 – self and peer evaluations) |
Student attended all workdays, made unique or exceptional
contributions to project, provided leadership, cooperated, demonstrated
skills in compromise and negotiation, shared responsibility and
resourcefulness, and participated with enthusiasm |
Student missed no more than one workday, participated in
project, cooperated, negotiated, shared responsibility for final product |
Student missed two or more workdays, failed to participate
and/or negatively impacted the group. |
|
Presentation has a central thesis, supported by valid
evidence (section 1 – organization/structure) |
Presentation has an obvious main thesis and connections are
clearly drawn between valid supporting evidence and the thesis |
Presentation has a thesis but it is not presentated in a
clear statement – or – cited evidence is incomplete in it’s scope,
or incompletely supports the thesis, or comes from questionable sources |
Presentation has no central thesis or has several theses or
indicates a thesis and doesn’t follow through, and/or evidence is
minimal (2 or fewer sources), unreliable, or only marginally supports the
topic |
|
Student synthesizes information and applies it in a
cohesive presentation (section 1 – content/knowledge) |
Reference to research sources is evident in the presentation and information that has been obtained from multiple sources is synthesized and applied in a thorough treatment of the topic |
Reference is made to sources of information but they are
either not diverse to begin with, or are not synthesized – and/or –
the relationship between the research and the topic is not clearly drawn
by the presenter |
No reference is made to sources of information – or –
no connection is drawn among the information sources or between the
information and the thesis |
|
Student’s presentation explores multiple facets of the
topic (section 1 – fair/factual) |
Presenters demonstrate that they have challenged themselves
by seeking out and accurately presenting diverse concepts or points of
view and attempting to reconcile them |
Presenters explore at least two facets of the topic but do
not identify or explore the more difficult or challenging arguments –
and/or – do not reconcile them or make a well-supported case for one
over another |
Presenters offer a one-sided or one-dimensional exploration
of the topic |
|
Student will be able to present information orally to an
audience familiar with the field of education, but not necessarily with
their specific topic (section 1 – quality of presentation) |
Presentation is clear and audible, there is minimal
reliance on notes, presenter makes eye contact with audience, has an
enthusiastic attitude and appropriate attire, and uses of the language of
the education field but without reliance on jargon |
Presentation volume or articulation are uneven or gestures
or posture |
Presentation was either unclear or inaudible or gestures or
posture were distracting, no little or no eye contact is made or read
significant portions of the speech. |
Return to Dana Franks's
Professional Interests Page
Return to Homepage for Dana Franks
Dana Franks, MLS
Faculty Librarian
Highline Community College Library, Des Moines, WA
Contact me at dfranks@hcc.ctc.edu
revised 06/15/06