Peer+Evaluation

So now is the time to decide.... Lets have a vote... put your name next to two suggestions and we will see which one gets the most votes. You can still add or change them if no one has voted for it. And you can change your vote if a new option comes up. We should be done voting at 4pm PST.

AERA 08 Session Title:
(8:45 PST, D-Day) Sue doesn't like her other suggestion, how about this one to more directly tie into the citizenship theme of the conference:

The 2008 theme is: Research on Schools, Neighborhoods, and Communities:Toward Civic Responsibility ( Title has a 15 words limit ) so we need to make a choice...

Sue: at 3:30 PST: We probably do not have time to go round with another suggestion, but just in case, (trying for consensus), how about:

Collaborative Knowledge Building in Communities: Research on Facilitating Peer Interaction and Critique
It has the communities word without the reified KBCs, the research emphasis, takes in all the papers, pedagogy will be implied, and dialog is implied. This would be my new first choice.

Jim: How about:

Collaborative Knowledge Building Communities: Peer Interaction in Online Learning
Sue: Don't like the 1st 3 words used together, too many adjectives to process Jim: yes, that's a big much - we could take David's suggestion, and make it: Collaborative Knowledge Building in Communities: Peer Interaction in Online learning

- The Pedagogy of Dialogue in Collaborative Online Learning: Urban planning for the new landscape
Margaret: I like the title but we don't do urban planning so I don't think this works; Sue: OK, but I think the Pedagogy of Dialogue part is really strong and under-addressed in literature.

-Knowledge Building Communities: The Pedagogy of Dialogue and the Role of Critique in Online Learning
Margaret:This was my attempt to get communites into the title, but stay close to our topic-(mmr first choice); Sue, I'm leaning to this as my first choice too, but do we need to keep clear that we are always just talking about course learning? Other kinds of neighborhood learning communities are knowledge-building too, so if our session addresses this, it's a great title. --David likes the phrase after the colon but is a little worried that 'communities' is hackneyed, despite its association with the Conference theme. Maybe "Knowledge Building in Community" which makes less of a claim and allows a bit more room than "communities"? (original wording is ok, too; david's first choice); Sue again: fine with any version of this as my second choice.

Jim: This is okay, but can we drop "The Pedagogy of" and "the Role of? ie. "Knowledge Building Communities: Dialogue and Critique in Online Learning" or David's "Knowledge Building in Communities: Dialogue and Critique in Online Learning" Ever since I learned that "Pedagogy" is derived from the term used for the Greek slaves that walked children to and from their lessons, I have a different stance to that word. The modern day "Pedagogues", by analogy, would be school bus drivers. ;-)

Knowledge Building Through Peer Critique: The Pedagogy of Dialogue in Collaborative Online Learning
David's new suggestion (7:30EDT, D-day)... Margaret Doesn't tie to the theme and it might be a bit too narrow to include all of the papers. David agrees.

The Pedagogy of Dialogue in Collaborative Online Learning: Blending Teacher and Learner Roles
David's less concrete suggestion Margaret: Fits all of the papers well (mmr second choice); Sue: I like this a lot, maybe second choice-especially if KBCs are too broad, but sort've don't like always seeing "teacher / learner roles" in titles. David likes the revision but agrees that it's not as strong as the current front runner. (David's second.)

Research on Student Roles, Peer Critique, and the Pedagogy of Dialogue in Collaborative Online Learning
Margaret: If I had a third vote it would be this one. Sue: this sounds like a laundry list of our papers.

Research on Collaborative Knowledge Building in Virtual Learning Communities
Sue's Suggestions for title Margaret agrees with David that virtual learning communities are different than online courses and we should be clear. Sue: How clear, and which title is most clear? David notes that 'virtual learning communities' has come to be associated with MUVEs and the like, so might mislead some in this context.

Research should be implied by AERA but sometimes it is good put it in the title.

[Jim, 1 Aug, 10:11am PDT): The 2008 AERA theme is: "Research on Schools, Neighborhoods, and Communities: Toward Civic Responsibility". So it might be good to have the word "communities" in our session title.]

Feel free to add comments or suggestions for the session or any of the individual paper. If you log in first, we will know who made the comments. If the proposal is accepted, we will indicate the time and place here.

Abstract
(119 word--must not exceed 120 words) Increasingly colleges and universities are experimenting with online, blended, and web-enhanced learning environments often with the goal of engaging students more actively in the learning process. Educators working from the social constructivist model of collaborative learning are exploiting the functionality of asynchronous learning networks to engage students in dialogic inquiry (Harasim, 2002; Hiltz & Goldman, 2005). The skills and abilities of students to provide support, feedback, and critique to their peers constitute an important dimension of this dialog. In this symposium we will share innovative research methods and findings from a range of different contexts with the goal of developing a deeper understanding of new roles of students as we study the pedagogy of dialogue in the online context.


 * Estimate Attendance** 75? 100?

Communities Distance Learning Learning Environments
 * Descriptors (choice from a set of terms)**

The Proposal
(total words 2640, not including references)

=[New Title Here]=

Increasingly colleges and universities are experimenting with online, blended, and web-enhanced learning environments often with the goal of engaging students more actively in the learning process. Educators working from the social constructivist model of collaborative learning, are exploiting the functionality of asynchronous learning networks to engage students in dialogic inquiry (Harasim, 2002; Hiltz & Goldman, 2005). Many instructors are moving part or all of their courses online without knowledge of how to structure learning experiences for dispersed learners, in a largely text-based environment. The skills and abilities of students to provide support, feedback, and critique to their peers constitute an important dimension of this dialog. In this symposium we will share innovative research methods and findings from a range of different contexts with the goal of developing a deeper understanding of the new roles of students and the shifting nature of learning in the online context.

In online learning, students are often required to organize thoughts into positions, address and debate multiple perspectives, and struggle with emerging understandings rather than receive positions fully formed by professors in lectures. Asynchronous learning networks can enhance collaborative learning and dialogic inquiry by extending the thinking time. Learners have the opportunity to reflect on a range of responses before formulating their own (Aviv 2002). In effect it is a slow motion conversation not only between teacher and students but from student to student (Bender, 2003).

Bransford, Brown, Cocking and the National Research Council (2000) signal the importance of formative feedback to learning. Schools often provide summative evaluation in the form of final papers, tests and grades, but what is missing in these educational contexts is the formative assessments which provide ongoing opportunities to improve the quality of thinking and understanding.

The quality of online learning relies on students' ability to engage one another in a way that leads to deeper understanding. The value of peer support and critique goes beyond assessment as it requires students to articulate positions that are not well formed, to make judgments about the ideas of others, and to develop interpersonal skills that help build emergent understandings. The ability to build knowledge collaboratively through dialogue supported by evidence is critical to disciplinary understanding (Harasim, 2002). Dialogic inquiry has the potential to deepen thinking (Bruffee, 1999). Collaborative dialogue that helps students engage in the process of building group knowledge becomes a potent facilitator of learning (Harasim, 2006; Bereiter, 2002).

Brown and Campione (1994) found that even young students can be an important force in providing feedback and supporting the learning of their peers. However, trust and a sense of intellectual camaraderie were necessary for students to "help one another solve problems by building on each other's knowledge, asking questions to clarify explanations, and suggesting avenues that would move the group toward its goal" (Brown and Campione, 1994, p. 25).

This symposium highlights the role of students in online learning through a focus on learning contexts in which students are asked to engage in collaborative dialogue. These papers extend what has already been reported about students roles in online learning in these areas:
 * Engaging in reflective, evidence-based argumentation about readings, investigations, or problems (Collison et al., 2000)
 * Offering constructive peer evaluations or feedback on individual or collaborative work (Hiltz, 1994; Riel et al., 2006)
 * Participating in case study discussions (Benbunan-Fich, 2002)
 * Advancing, moderating and assessing the quality of the discourse in online seminars (Harasim, 2006)
 * Creating new materials to develop their group understanding (Hiltz & Goldman, 2005)

Students who move to learning online are often unprepared for the blended roles of learner and teacher. They doubt the authority of their ideas and are reluctant to build ideas without the direction of a recognized expert who can point them in the direction of the accepted ideas (Bruffee, 1999). They are, in fact, more ready to place value on the ideas of their peers than to see the value in their own ideas (Riel, Rhoads, & Ellis, 2006). This makes it difficult for students to engage in a process of thinking built on open debate and thoughtful critique.

Collectively, the papers in this session will provide a deeper understanding of the potential as well as the challenges of asynchronous collaborative dialogue and strategies for deeper online learning. The symposium will include a discussant-researcher who has designed and studied online learning environments. The session will conclude with audience participation in order to frame next steps for research in emergent technology teaching and learning.

(Stone Wiske, Harvard University, as the Discussant)

Paper Abstracts
Sarah Haavind (339)
 * Facilitating Cognitive Presence to Support and Deepen Distributed Cognition through Online Learning**

In E-Learning in the 21st Century (2003) Garrison and Anderson define “cognitive presence” as, “the intellectual environment that supports sustained critical discourse and higher-order knowledge acquisition and application” (p. 55). The term clearly captures the sense of the valuable contribution asynchronous dialogue is capable of supporting: sustained critical discourse. However, fostering such postings from learners, or supporting instructors in easily facilitating cognitive presence, are challenges that remain elusive to learners, instructors and to researchers.

In a more recent study of cognitive presence (2005) Garrison and Cleveland-Innes showed that in order for interaction to progress to deepen ed levels of dialogue and learning, course design and teacher leadership must incorporate actions and elements that promote collaborative dialogue among learners. This study supports the Garrision & Cleveland-Innes finding that structure and leadership are “crucial for online learners to take a deep and meaningful approach to learning” (p. 133) and describes effective approaches in the context of secondary online learners.

Building on an earlier study (Haavind, 2006), the current study examined three, highly collaborative Virtual High School courses using both qualitative and quantitative measures. The pedagogical objective in all three of these cases was to enhance the core learning activity, collaborative dialogue, in order to establish a clear sense of cognitive presence among class members. Analysis revealed both course design elements and teacher leadership approaches that promoted cognitive presence in the form of sustained collaborative dialogue.

The analysis of all three cases suggest that learners need explicit instruction on how to collaborate in an asynchronous, text-based environment. Strategies that were abstracted include private feedback on the collaborative quality of learner postings; offering constructive suggestions; using task design elements to directly scaffold collaborative dialogue; and the design of activities that required graduated levels of interdependency among class members over time.

This research draws a connection between activities that involve explicit teaching of how to collaborate and sustained collaborative dialogue. Teacher leadership and activity designs that show students how to collaborate serve to leverage learners to a more reflective stance in their participation in asynchronous dialogue.

**Support, Review, Extensions and Critique**
Margaret Riel and Sue Wolff (360)

Much of schooling requires students to acquire knowledge alone with limited opportunities for group inquiry, interpretation, design, and critique (Dewey, 1938). Students come to the university with the expectation that they will learn from experts—their professors—rather than from one other (Bransford, Brown and Cocking, 1999). Additionally, students have been conditioned to think of evaluative feedback as personal and private from instructor to student (Webb & Palincsar, 1996). Shared public critique represents a significant shift in the learning context facilitated by online learning contexts (Harasim, 2002, 2006). Peer critique necessitates a reacculturation of students to help them trust their authority to evaluate the work of others and to develop a respect for the value of peers insights (Bruffee, 1999). Moving from knowledge acquisition to actively building knowledge is difficult for students because it confronts the basic norms about schooling (Roberts, 2004). This study examines a process of having students provide feedback to their peers engaged in action research. The research question is: How does the process of coding one's previously given feedback shape the feedback one gives to peers in the future?

The context of the study is two cohorts of students (N=16 in each group) in year-long course on action research. The students received feedback on their work in learning circles of 4-5 students from the professor and were also instructed to give each other feedback. Midyear, one cohort of students evaluated the forms of feedback they had provided to their peers using a rubric derived from a discourse coding method (Riel, Rhoads, & Ellis, 2006). The student feedback given in the last semester will be compared with the cohort who experienced a similar context but did not code their feedback.

Previous research found that students gradually begin to display tendencies toward collaborative critique when it is modeled for them by the professor. Left unclear is whether this gradual tendency toward trust and collaboration could be accelerated by a more structured emphasis on its value (Riel, Rhoads, & Ellis, 2006). This research seeks to more clearly determine whether a reflective awareness on the part of the students about their communication with peers yielded more significant results than modeling assessment.


 * The global shape of online interaction: Weaving multiple coordinated threads for learning**

James Levin (344) University of California, San Diego

New educational media allow for new kinds of educational interaction (Hiltz & Goldman, 2005), including innovative forms of peer interaction (Harasim, 2002). This paper reports on several kinds of analyses of online interaction that occurred as parts of several graduate education courses, interaction which occurred through multiple threads of discourse (Black, Levin, Mehan, & Quinn, 1983). These analyses support the concept of a "global shape" of an interaction, with "width" that is the number of multiple simultaneous topic threads, and "mean thread length" measured by the average number of turns involved in each topic thread. These analyses also support a measure of "interactional influence" of a participant, which is the percentage of total interactional turns that are contained in topic threads that the participant started. The "wide" parts of the interaction are more likely to be marked by "trouble talk", explicit expressions by the participants of difficulties with following the interaction.

The analyses presented here show some of the upper limits of interactivity in online education. If an interaction is too interactive, then it can overwhelm the processing capabilities of at least some of the participants. While highly interactive learning environments are valuable for supporting learning, they also come with a cost. The analyses here suggest that interactivity should be carefully chosen as a tool for promoting learning, and that the design of learning environments take into account both the benefits and the costs of interactivity. The analyses also shed light on the important role of multiple kinds of mediators in creating and sustaining productive educational interaction, provided by both teachers and students.

The research reported in this paper provides a deeper understanding of the role of interactivity in education, both online and more generally, describing the ways that multiple coordinated threads of discourse can be woven together to create productive environments for learning. It also points to the critical role of mediators in coordinating the distributed nature of learning across space and time and across people and media.

Building Knowledge through Cycles of Revision: Peer Assessment in a Blended Environment for Professional Learning
David Eddy Spicer Harvard Graduate School of Education (346)

Online collaborative learning has become a vital complement to face-to-face approaches for job-embedded teacher professional development, defining new possibilities for teacher community and for sustaining inquiry into instructional practice (Barab & Duffy, 2000; Dede, 2004; Wiske & Perkins, 2005). The realization of such potential, however, hinges on the fit between the affordances of technology and existing social structures (Bielaczyc, 2006; Fusco & Schlager, 2004). Such a fit is especially critical across the diverse participant structures that constitute blended learning environments combining online, asynchronous interaction with face-to-face meetings.

This comparative case study examines the role of peer assessment in revising units of curriculum generated by school-based teams in a blended course on the Teaching for Understanding framework (Blythe, 1998; Wiske, 1998). The study examines (a) how support and challenge in peer assessment expand or constrain cycles of revision and (b) how the exercise of expertise in interaction sustains or closes down inquiry. Nine teams, each of 3-4 teachers, in 3 schools were selectively sampled from participants in a 12-week blended course. Data collected include documents from weekly on-site team meetings, email messages, and asynchronous discussion threads among team members, between teams, and with an online coach. The analytic framework draws on social semiotic approaches to multimodal discourse analysis (Baldry & Thibault, 2006; Eggins & Slade, 1997; Halliday & Matthiessen, 2004; O'Halloran, 2004), exploring interpersonal meaning making through close analyses of speech function and speech role (Eddy Spicer, 2006).

Initial findings trace the role played by //ideational authority// —status bestowed by others in the group around one’s ability to serve as a source of authoritative information—to explain differences in the impact of peer assessment on teachers’ collective efforts. The mediational role played by representational artifacts hinges on the ways ideational authority manifests itself in representations and in interaction. The study also seeks to explain how certain teams take the representational artifact as an “improvable object” (Bereiter & Scardamalia, 1996), that is, a focus of generative group interaction and revision, while similar artifacts to other teams appears as canonical, unavailable to critique and question.

References

Aviv, R. (2002). Educational performance of aln via content analysis. //Journal of Asynchronous Learning Networks// Retrieved March 25, 2005, from http://www.sloan-c.org/publications/jaln/v4n2/index_member.asp Baldry, A., & Thibault, P. J. (2006). //Multimodal transcription and text analysis: A multimedia toolkit and coursebook.// London; Oakville, CT: Equinox Publishing. Barab, S. A., & Duffy, T. M. (2000). From practice fields to communities of practice. In D. H. Jonassen & S. M. Land (Eds.), //Theoretical foundations of learning environments// (pp. 25-55). Mahwah, NJ: Lawrence Erlbaum Associates. Benbunan-Fich, R. (2002). Improving education and training with information technology. //Communications of the ACM, 45//(6), 94-99. Bender, T. (2003). //Discussion-based online teaching to enhance student learning.// Sterling, VA: Stylus Publishing, LLC. Bereiter, C. (2002). //Education and mind in the knowledge age.// Mahwah, NJ: Lawrence Erlbaum Associates. (preface)(first chapter) Black, S. D., Levin, J. A., Mehan, H., & Quinn, C. N. (1983). Real and non-real time interaction: Unraveling multiple threads of discourse. //Discourse Processes, 6,// 59-75. Bielaczyc, K. (2006). Designing social infrastructure: Critical issues in creating learning environments with technology. //Journal of the Learning Sciences, 15//(3), 301-329. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). //How people learn: Bridging research and practice.// Washington, D. C.: National Academy Press. Brown, A., & Campione, J. (1994 ). Guided discovery in a community of learners. In K. McGilly, //Classroom lessons: Integrating cognitive theory and classroom practice.// Cambridge, MA: MIT Press. Bruffee, K. A. (1999). //Collaborative learning: Higher education, interdependence, and the authority of knowledge.// Baltimore: John Hopkins University Press. Dede, C. (2004). Distributed-learning communities as a model for educating teachers. Retrieved April 6, 2004, from http://www.gse.harvard.edu/%7Ededech/505/Distributed-Learning_Communities.pdf Eddy Spicer, D. (2006). //Collective inquiry in the context of school-wide reform: Exploring science curriculum and instruction through team-based professional development.// Unpublished Doctoral dissertation, Harvard University, Cambridge, MA. Eggins, S., & Slade, D. (1997). //Analysing casual conversation.// London, Washington, D.C.: Cassell. Fusco, J., & Schlager, M. (2004). Teacher professional development, technology, and communities of practice: Are we putting the cart before the horse? In S. Barab, R. Kling & J. Gray (Eds.), //Designing virtual communities in the service of learning.// Cambridge, UK: Cambridge University Press. Garrison, D. R., & Anderson, T. (2003). //E-learning in the 21st century: A framework for research and practice.// London and New York: Routledge-Falmer. Garrison, D. R., & Cleveland-Innes, M. (2005). Facilitating cognitive presence in online learning: Interaction is not enough. //The American Journal of Distance Education, 19//(3), 133-148. Haavind, S. (2006). //Tapping online dialogue for learning: A grounded theory approach to identifying key heuristics that promote collaborative dialgoue among secondary online learners.// Harvard University, Cambridge, MA. Halliday, M. A. K., & Matthiessen, C. (2004). //An introduction to functional grammar// (3rd ed.). London: Arnold. Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning. //Internet & Higher Education, 3//(1-2), 41-61. Harasim, L. (2002). What makes online learning communities successful? The role of collaborative learning in social and intellectual development! In C. Vrasidas & G. V. Glass (Eds.), //Distance education and distributed learning.// Greenwich, CT: Information Age Press. Harasim, L. (2006). Assessing online collaborative learning: A theory, methodology and toolset. In B. Khan (Ed.) //Flexible Learning in an Information Society.// Hershey, PA: Idea Group Publishing. Hiltz, S. R. (1994). //The virtual classroom: Learning without limits via computer networks.// Norwood, NJ: Ablex Publishing Corporation. Hiltz, S. R., & Goldman, R. (Eds.). (2005). //Learning together online: Research on asynchronous learning networks.// Mahwah, NJ: Lawrence Earlbaum Associates. O'Halloran, K. L. (2004). //Multimodal discourse analysis: Systemic-functional perspectives.// London; New York: Continuum. Riel, M., Rhodes, J., & Ellis, E. (2006). Culture of critique: Online learning circles and peer review in graduate education. In T. Roberts (Ed.) //Self, peer and group assessment in E-learning.// Hershey, PA: Information Science Publishing. Roberts, T. (2003) //Online collaborative learning: Theory and practice//. Hershey, PA: Information Science Publishing.. Rumble, G. (2001). Re-inventing distance education, 1971-2000. //International Journal of Lifelong Education, 20//(1/2), 31-43. Wiske, M. S. (Ed.). (1998). Teaching for understanding. San Francisco: Jossey-Bass. Wiske, M. S., & Perkins, D. N. (2005). Dewey goes digital: The wide world of online professional development. In C. Dede, J. P. Honan & L. C. Peters (Eds.), //Scaling up success: Lessons learned from technology-based educational innovation.// San Francisco: Jossey-Bass. Webb, N. M., & Palincsar, A. S. (1996) Group processes in the classroom. In D, Berliner & R. Calfee (Eds.), //Handbook of educational psychology//. New York: Macmillan.