Masterclass

Learning Objectives

The Peergrade Masterclass will explain the concept of peer evaluations and aid you when integrating peer evaluations in your courses. The learning objectives are:

The course spans 4 lectures each lasting about 15 mins. Get the whole learning experience in one seating or explore the lectures individually.

If you have question or input for the content, please use the comment section at the end of each lecture.

 

Course Outline [will be updated when the outlined is final]

The course is divided into four sections plus introduction, outlines below.  The total duration of the course is approx. one hour, and you can take it in one seating or come back for individual sections. Each section concludes with an "In Action" pdf with the main takeaways for download or print.

Introduction Learning objectives
  Course structure
  References
The big picture:  Background
  Setting the scene: Peer evaluation as a method to reinforce learning
  Benefits
  Limitations & challenges
Integrating peer evaluation in your courses Combining with other activities
  Security
  Motivating students
  Potential pitfalls
Designing for good feedback The importance of rubrics
  General guidelines for good rubrics
  Making effective rubrics
Validity in the numbers Scaffolding to avoid bias
  Ensuring hgh-quality feedback
  Use in grading

 

Who is this course relevant for?

The content of this course will be relevant for teachers, schools and academic organisations.

Teachers

Schools and academic organisations considering an institution- or department-wide implementation of peer evaluations, want solid knowsledge of the associated implications. As with any change in the teaching environment, centrally organised guidelines and best-practices can aid the effectiveness of new tools and practices. In the light of common knowledge and access, first-movers and sceptics can benefit from sharing opposing views. This course will provide the responsible personnel with a strong foundation for formulating how peer evaluation is best utilized in the individual organisation.

Students might also take an interest in this course. Most likely students will browse the Masterclass after being invited to use it by a teacher. It could also be that the student wants to encourage the use of peer evaluations where he is currently enrolled. Understanding the theoretical backround of peer evaluation  is certainly valuable in increasing the student's own learning outcome and performance.

The big picture

Definition

“Peer assessment is an arrangement for learners to consider and specify the level, value, or quality of a product or performance of other equal-status learners. Products to be assessed can include writing, oral presentations, portfolios, test performance, or other skilled behaviors” [Topping, 2009]

An arrangement suggest a structured approach, which typically incorporates one or more digital tools that can ensure quality and lessen the teacher's burden of facilitation. We present popular tools in lecture x.
Considering is a key word in the definition, as it suggest the feedback provider will reflect and evaluate - important aspects in high level learning. By specifying one's reflections, the peer is presented with alternative viewpoints - another important component in learning - while the teacher receives input for both course modification and grading. Equal-status learners is the core of peer assessment, and entails feedback communicated on a common basis and in a familiar language.

'Peer assessment' is the term used in this course. Several synonyms are often used; peer feedback, peer evaluation, peer grading - though this last one can entail a more finite form of input.

The need for feedback

The common theoretical progression in learning objectives, as presented in Bloom's Taxonomy, defines the order of thinking skills a student possesses. For real-world application of knowledge, the aim of any teaching effort is to take the student from knowledge to evaluation. High-level learning, particular evaluation, requires feedback and is thus constrained by the quality and quantity of teacher-student or student-student interactions.

Giving feedback is an individual task and rarely possible for teachers due to limited time. When handled by teachers and instructors, it also means evaluations are at a significant expense for the organization. Thus, Feedback quality starts to suffer due to budget constraints. 
Larger class sizes and versatility among students are factors that reinforce the challenges.

The disharmony is clear on a pedagogical level, but is it also acknowledged by the students? A 2016 survey of students (https://medium.com/peergrade-io/what-kind-of-feedback-do-students-get-3f796d00c422#.1piqw8mio) showed that 64% of students get some kind of feedback on hand-ins. Of those, 40% actually have to ask for the feedback and 37% have problems understanding the feedback.
There have been several examples of students protesting the lack of feedback. Frustrated Danish students have put Batman quotes and food recipes (https://medium.com/peergrade-io/budget-cuts-and-batman-quotes-c25112011426#.lwan4bw1k into their hand-in's, thereby confirming the teachers were not reviewing assignments.

To sum up: There is a very evident gap between desired outcome and available resources.

Before going more into depth with peer evaluation, it is sensible to take a view on some trends in educational settings in the western world as well as an outlook into the future of Educational Technology (EdTech). Education is generally a stable industry. When it comes to utilizing state-of-the-art technology, the industry is rather conservative.

Enter peer evaluation

Peer evaluation is meant as a method to close the gap by ensuring all students can achieve evaluation, independent of class size and composition. In the process of peer evaluation, a single student will take on the role as both evaluator and evaluatee including several aspects on the Analysis-Synthesis-Evaluation levels:

  1. Assess the performances of others
  2. Compare the work of others to one's own work
  3. Reflect on one's own work on account of received feedback
  4. Consider alternative solutions for revised or future assignments

Benefits

More and better feedback

Peer grading provides students with the feedback they deserve. When grading a large number of assignments, the result is often that students receive very little feedback on their work. The best way to learn is to write and solve problems, but without feedback this becomes unsatisfactory.

Learn on a higher level

Spending time on evaluating and giving feedback to the work of others forces students to reflect on the key learnings of the course. This task is an exercise of the highest step in Bloom's Taxonomy: Evaluation and elevates student learning.

Save Time

An immense amount of time and money is spent by teachers and teaching assistants grading assignments. Peer grading offers a way to dramatically reduce the time spent on grading while actually increasing the quality of the feedback.

Timely feedback to teachers

How are the students doing in my current course? What degree of involvement are students showing? 

Such questions are fundamental when a teacher wants to improve a course between semesters or adjust lectures during an ongoing course, based on and involvement of students. Many teachers receive little feedback, and have to make course improvements largely based on guesswork. Peer evaluation can be a very effective source of teacher feedback, as the data can be automated (assuming a digital tool is used).

Communication & collaboration skills

Students can be encouraged to communication academic topics in a language that is appealing to peers instead of just the academic expert (teacher). This is a valuable learning in itself. This learning can be reinforced by allowing students to hand in alternative formats - something that is often impractical when the teacher is responsible for handling all feedback. We have used video in one course. Read about the outcome here:  https://medium.com/peergrade-io/making-students-create-public-tutorial-videos-as-a-course-assignment-33e918078fe5#.t9etynrat

Current trends in education

Part of the big picture is the fact that peer assessment is correlated with current trends in educational settings.

Class sizes are growing in many countries. The individual teacher will have to be more efficient to avoid diminishing educational quality. Efficiency through standardization is not the entire answer.

E-learning is currently gaining traction, which facilitates the development of new learning tools and formats. Increased usage of e-learning creates a good basis for integrating modern tools in courses.

Collaborative and interdisciplinary skills are becoming a more integral part of many studies, to a large part driven by market needs. Therefore, many assignments are now handled in groups.

Budget cuts are enforced in many countries - particularly those with a mature and socialized educational system. In Denmark, for example, education funding will be reduced by 8.7B DKK (US$ 1.3B) over the next four years. In Finland the overall budget cuts on higher education is evaluated to be around €500 million (US$556 million). An immediate effect of budget cuts is elimination of certain programs, thus increased specialization.

Limitations

When talking about benefits, we must also address limitations, in order to further identify what peer evaluation is - and what it is not.

Direct grading of students

Peer evaluation is not about delegating grading. The final grading of students is a task requiring expert knowledge and experience. In other words, it is a task for teachers. Empowering students to provide a final grading is not recommended. In fact, it is illegal in some countries.

Data from peer evaluation sessions is not directly transferable as grading's. It is, however, logical to include peer the outcome of peer evaluation as one component in grading. Particularly, two aspects can be considered relevant - the feedback received as an indication of performance, and the feedback given as a measure of involvement.

Requirements for teacher involvement

The teacher should not expect that peer assessment can be handled by the students alone. To ensure targeted and high-quality feedback, the teacher involvement is important. Another crucial factor is motivating students. They will need to spend a significant amount of time, and the purpose is not always apparent to them. These aspects are covered in later sections.

The big picture in Action

“Peer assessment is an arrangement for learners to consider and specify the level, value, or quality of a product or performance of other equal-status learners. Products to be assessed can include writing, oral presentations, portfolios, test performance, or other skilled behaviors”

Peer evaluation closes the gap between required high-level learning and available staff resources, by allowing student to:

  1. Assess the performances of others
  2. Compare the work of others to one's own work
  3. Reflect on one's own work on account of received feedback
  4. Consider alternative solutions for revised or future assignments

The benefits of peer evaluation

  • More and better feedback
  • Learn on a higher level
  • Time savings
  • Timely feedback to teachers
  • Communication & collaboration skills

Before investing in technology for peer evaluation, consider these aspects:

  • Where is the need? 
  • Are the teachers on board? 
  • How does it complement the teacher's activities? 
  • How will the students benefit? 
  • What is the implementation plan? 

Integrating Peer Evaluation in your courses

The importance of integrating

Peer evaluation is not an isolated activity. It is closely tied to learning objectives, assignments and evaluation of student performance. As such, it is important to handle it as an integrated part of the course in line with readings, assignments, group work, etc.
In order to effectively integrate peer grading into your course does require you to rethink the way you teach and assign projects to your students. But once you have taken the time to integrate it into your teaching, it will give you the possibility to use your time more constructively as well as offer your students the possibility to learn a valuable skill that can be used outside the academia.

Consider this metaphor ...

In this exercise the person has to take a number of steps in order to . The person can retain balance by using the robes. Without these he would easily roll off and fall to the ground. The robes are not fail-safe, so there is also a harness that catches the person and ensures he can return safely to the exercise in case he falls. Now consider this - the harness is not merely a safety feature, but also what encouraged the person to even engage in this exercise.
In this metaphor, the ropes are the scaffolding and the safety harness is encouragement by meaningful course integration.

Meaningful integration

What goes here?

Set expectations

Make sure the students know how you expect them to use peer evaluation in your course. Is it mandatory to participate, how many assignments and students will they provide feedback to, how will peer evaluations influence the final grading, etc.

If a service provider is used to handle the feedback, then also make sure the students know what challenges the service provider can handle (e.g. technical issues) and what issues are addressed in class (e.g. uncertainty regarding evaluation criteria).

Use feedback actively

The individual student will benefit directly from considering feedback given and received. However, even greater value can come from feedback being used activily in solving .

Share insights

When reviewing the total sum of feedback on an assignment, it will become apparent what criteria the students are over- and under-performing on. This is a valuable insight for both teacher and students when choosing topics to focus on throughout a course.

By sharing insights into feedback performances of different (unnamed) student, the teacher can coach the student on how to give better feedback.

Giving feedback

If students know about good feedback, it significantly increases the value created during peer assessment. It is also worthwhile for the teacher to consider good practices for giving feedback for several reasons. First, good feedback is the cornerstone of successful peer evaluation. Second, it is a great collaborative learning experience the  students. Thirdly, knowing about good feedback makes it easier for the teacher to design the rubrics.

Let us first take a look at some tips for giving good feedback. You can download a handy pdf with the tips, and share it with your students. Afterwards, we will consider individual versus group feedback.

5 tips for giving good feedback

Giving feedback can be tricky. It can start conflicts, but it can also provide direction and spark epiphanies. Here we present five ways in which you can give effective, focused, and well-grounded written feedback.

1 . Stay Focused

Have you ever been given feedback along the lines of “Adequate,” “Good,” “Confusing” or “Your sentences start weirdly.” These are all examples of unfocused feedback that are difficult to understand and convert into action.

When giving feedback, you want to provide clear direction through actionable and text-specific comments instead. For instance, “What’s your main point here?”, “If you disagree, put that idea up front and explain,” “Consider integrating these ideas” or “Be more specific. Say where and when.”

You should make sure that you comment on both what is important and what the person receiving the feedback is able to control.

2. Be Timely

There are few things more annoying than being told “You should have done this” or “If you had included this essential point it would have been perfect” once you have already concluded a course or handed in the final assignment. Rather than at the end, feedback given along the journey of learning provides a better understanding of the subject matter when you need it most and enables you to make changes before it is too late. Using tools like peergrade.iocan help to make the feedback timely and current.

When it comes to peer feedback it is important to provide it before you and your peer forget the subject matter, before other assignments requires your attention, and before the potential learning outcome becomes obsolete.

3. Create A Positive Experience

I think we can all agree that the goal of feedback is increasing the learning outcome and development of the recipient of feedback. Being clear in your feedback seems obvious, but maintaining a positive experience for the receiver of feedback is easy to overlook and also harder to get right.

It is necessary to understand that the purpose of giving feedback is not to drag anyone down; it is not about pointing out flaws, but rather seeing possibilities of improvement and whether or not you are on the right track. This is not the same as being overly positive and only focusing on good things. But creating a comfortable atmosphere of trust and respect which helps with delivering difficult feedback and also to make sure that the feedback is listened to and accepted. Writing things like “You showed me how well you understand the model when you explained how it helped you structure your data and gain valuable insight,” makes it really clear where you find the ‘good stuff’ and what is so great about it.

Feedback is also a balancing act and can only be effective when it is given in a constructive and encouraging manner. Speaking your mind and claiming statements like “This is absolutely terrible! Have you even put any effort into this?” will not have any beneficial value neither to you nor the person receiving your feedback.

It is important to avoid sarcasm and use simple (understandable) language in your feedback, as the person receiving the feedback is more likely to accept and learn from your comments.

4. Be Specific

— Provide Specific Suggestions About What To Do Next

When giving feedback, it is equally important to present specific suggestions on how the person could continue improving the assignment and subject matter. Think about what steps the person should take to move their work to the next level.

Reviewing someone’s work and providing them with possible solutions to approaching the topic will motivate the person and further their learning experience.

5. Look At Things From Different Angles

— Using De Bono’s Thinking Hats

De Bono suggests using the metaphor of colored hats to think from different perspectives, focusing on one perspective, or one hat, at a time. For example, put on the yellow hat and think only about the good things in the peers work. Putting on the green will help you focus on creativity and look for new and different ways your peer can approach the assignment. While the black hat will encourage careful evaluation, taking a hard look at the potential weaknesses to an argument.

 

Giving feedback as a group is not as effective as doing it alone

How can the educational system improve peer feedback, so that students can get both a better learning experience and receive thorough feedback? This post takes a closer look at the dynamics of using peer feedback individually and in groups.

The success of improving a student’s learning experience through peer grading and peer feedback is highly dependent on the willingness of students to give and receive constructive feedback. When it works, the method can offer a beneficial learning experience for all parts. One question we often talk to teachers about, is whether giving feedback as individuals or as groups works best.

Our experience, both as teachers and as the facilitators of peer evaluation processes in many courses through Peergrade is that allowing students to give feedback individually has the biggest learning potential. Today many teaching activities are completed in groups, from the early years of school until the completion of university degrees. Learning to work in group settings is an essential part of education.

But we believe that a combination of group-based work and individual work gives the best outcomes. Completing assignments and solving tasks together, followed up by individually giving feedback and assessing work of others is a great combination. This offers the students a greater possibility to engage with the material and optimise their own learning. This is not to claim that assignments should not be completed as a group, as there are definite advantages to such a working method both for the students as well as for the teachers. It is rather to suggest that joint feedback giving will not necessarily have the same educational value to one’s learning, as when students approach the project / task individually.

Similar conclusions were drawn by Pozzi et al. (2016)[1] when reviewing the potential of online peer grading. In their pilot study, the researchers conducted a qualitative and quantitative comparison of the work dynamics of dyads (pairs of students) and groups of five or more people. What their findings suggest is that peer grading can be a more successful teaching method when used in smaller groups, as pairs showed to be more active in taking responsibility for their own learning.

Looking at the differences between the two different social structures adopted (groups and dyads), it seems that dyads worked better: people working in pairs were more active as to Individual (C1), as well as Group (C2) knowledge building. (Pozzi et al. 2016, 101)

Pozzi et al. (Ibid.) findings additionally concluded that:

Dyads members were also forced to play more the teaching role and it seems that a stronger effort for organizing the work and facilitating the discourse was required to learners in pairs.

Most teachers that have worked with group work in their classes will know that while most students benefit in many ways from the experience, some do not. The potential of “free-riding” — either by choice or as an involuntary consequence of the other members of the group — can lead to less engagement, less interaction with the material and consequently less learning.

Peer grading is a collaborative effort and it is important to explore the optimal ways in which to engage all the students in order to improve their learning experience. Combining group based problem solving with individualised peer evaluations and peer feedback gives “the best of both worlds” while not requiring a huge time investment from the teacher.

 

Motivating students*

Motivating students can be difficult but it gets even more complicated when new technology is in the mix. We will look at some basic advice for student motivation and how we can apply these to motivating students in using new methods and technology in the classroom. Change initially evoke feelings of discomfort and resistance among some people. Therefore, we conclude this section with a look at the skeptical student.

Intrinsic and Extrinsic motivation

In order to understand how to motivate students it is important to touch briefly on the basics of motivation; student motivation can be either intrinsic or extrinsic and there are advantages and challenges to both. The former comes from within the student and their natural interest in a topic of study, while the latter relates to outside factors, such as living up to expectations or earning a good grade.

While in higher education, students are often intrinsically motivated, having chosen their course of study based on their own self interest. But sometimes university students still need help and encouragement to find their motivation, especially when there are faced with incorporating new technology and methods into their learning.

How to motivate students?

A purpose they can relate to

Just as a student’s passion for a particular topic can motivate you as a teacher, you can also inspire your students to become more engaged by expressing your own interest in a topic. This also crosses over into displaying a passion and belief in using new technology in the classroom. Don’t just tell your students that they have to peer evaluate and hand them a generic text on peer evaluations. Rather, explain why they are doing it in your course, so they can relate. Follow up the purpose with a detailed explanation of how it is conducted and what tool(s) will be used - if you don't plan on explaining this yourself, make sure the students have access to a short  and relevant guide.   - explain why, what value you expect it to create, and exactly how it is done.

Personalized Learning

It’s not just about being personal but also about personalizing the educational experience. Giving students an opportunity to choose their own topics or form of assessment will increase their motivation. Many edtech solutions were created just for this reason, so take advantage of the opportunity to let students personalize their education. For example, letting students upload videos, links or files making it easier for you to allow students to explore different mediums. The motivation and desire to use whatever technology you introduce will increase if student’s know it will allow them to personalize their education.

Make it relevant

Similar to being personal with students, it is important that student’s understand the “what” and “why” of their education. It can be helpful to go through the learning outcomes with the students and make clear the expectations for the course and the assignments. This is why we love using rubrics at Peergrade, they are a great form of communication between student and teacher that clarify expectations and objectives. They also answer the questions about “why am I doing this assignment?” and “What am I supposed to learn from this assignment?”.

Student’s will have these same questions about using technology and tools you introduce. “Why am I using this?” and “What is the benefit for me?”. Be clear and help your student’s understand the benefits for them as students.

Motivation through use

While praise can boost the students confidence, constructive criticism can be the motivating force for a student to keep improving and pushing themselves. We like to talk about all the great benefits to using Peergrade and getting feedback, and motivation is not often one of these benefits discussed. But feedback can be the fuel to keep a students motivation going, letting them know what they have done well and what they can aim to improve. 

Educate and evaluate

Giving and receiving feedback is often unknown territory for students. The teacher should spend a bit of time introducing how to give and use feedback (explained in the previous subsection). By doing so, students will feel less estranged while the entire process will appear more meaningful.

Furthermore, students should receive feedback on their peer evaluation performance during the course, which implies routines should be measurable. This is easily obtained with a good tool in the digital realm, while an "analogue" approach will require the students to report on the quantity and quality of feedback - an administrative procedure that is often discouraging.

Since you have established a clear purpose for peer evaluations, you can easily empower the students with assessing the performance of peer evaluation activities, as part of the current course evaluations. Take this opportunity in order to address issues and concerns, while strengthening the integration of peer evaluations in your courses.

 

Peer Evaluations and the Skeptical Student

Peer evaluations encourage students to think critically and become active users in the feedback loop. With all the benefits peer evaluations provide, it is useful to learn how to tackle some of the main concerns that students have in relation to using peer evaluation in their coursework and how you can address and stop the skepticism before it even starts.

What skepticism?

These concerns are not pulled from thin air, we have analyzed feedback we have received from students after they have used Peergrade for one semester. Across students, we have seen that there are common concerns with peer grading and peer evaluation including: lack of teacher involvement, extra course work and feeling unprepared to give feedback. Below are three ways that these concerned can be addressed in the classroom.

Question of Integration

One of the main concerns students have had with using peer evaluation is related to the way it has been incorporated into their course work. If students already have a long list of assignments they need to complete for the course, then adding peer evaluation might feel like extra work, because it feels just like what is, an addition. If students are going to engage and put proper effort into peer evaluations, they need to feel that it is an important part of the course material.

Teacher Involvement

It is necessary to make sure that students know that peer evaluation does not mean that teachers are removing themselves from the task of teaching or grading. Students need to be assured that their teachers are equally involved in peer evaluation by being there to guide them, whether through clear evaluation rubrics or in class instructions and by bringing some of the discussions up in the classroom.

The Importance of Rubrics

Finally, we cannot write enough about the importance of rubrics, without clear guidelines and instructions, students are not able to offer constructive feedback to their peers. A good rubric takes time and consideration, but the time spent will pay off when students are able to fairly and reliably assess their peers and provide helpful feedback.

So where does that leave us?

The suitability of anything new in the classroom often comes down to how well its been implemented and integrated into the course work. This will determine both the usefulness of the new tool for the teacher and how well students will accept the change.

In order to effectively integrate peer grading into your course does require you to rethink the way you teach and assign projects to your students. But once you have taken the time to integrate it into your teaching, it will give you the possibility to use your time more constructively as well as offer your students the possibility to learn a valuable skill that can be used outside the academia.

Exercise: 

Fill in the blanks in this peer assessment guide, to set the expectations between you and the students. You can then export and share the document.

Peergrade is 

Peergrade is used for:

Time spent:

Influence on grading: 

Potential pitfalls

Students are overburdened or unmotivated, resulting in diminishing quality

Students do not have the required expertise to answer certain questions.

Limiting formats and problematic files ...  https://medium.com/peergrade-io/how-we-handle-problematic-files-in-peergrade-io-7add8626d359#.ytebnel1h

EdTech in the classroms / The rise of Edtech

Edtech is perceived as part of the answer to - and part of the reason for - the aforementioned trends. Much of teaching and learning has already moved online and there are a number of great Edtech companies and innovators striving to impact education. Massive Open Online Courses(MOOCs) are making learning accessible in every corner of the world and partnerships with major institutions allow people a top notch education at more reasonable prices.
According to Techcrunch, global investment in edtech will reach $252 billion by 2020. Even companies like Amazon are sensing the potential in edtech as they are preparing to launch Amazon Inspire, a free platform for educators to search and share digital educational resources.

There are good reasons behind why edtech is gaining new interest and investors. Firstly, edtech allows for education to become personalized, accessible and cheap. Massive open online courses (MOOCs) have been the biggest edtech trend recently with companies like Udacity and Coursera offering nanodegrees and partnerships with many major universities. Many of these MOOCs offer free courses or significantly cheaper options that allow people to continue their education or just learn out of pure curiosity.

Secondly, education is a relatively safe investment because it is a huge market that has gone relatively unchanged (and unsaturated) over the last century. While technology has always made it’s way into classrooms, the fundamentals of teaching have not changed greatly to incorporate these new technologies. There is still room for edtech to positively impact the classroom.

Finally, there is also the inevitable behind edtech, students will be expected to use technology in daily life, both for leisure and future careers, so integration should begin in the classrooms. Integration is the reason edtech has failed time and again. For edtech to thrive in any classroom and at any educational level, it needs to come with proper training and integration. In an interview with Edsurge, Alan November perfectly labels technology that is not implemented correctly a “$1000 pencil”. Smartboards have to be the best example of this “$1000 pencil”, they were hyped across schools as the next big thing, promising teachers and administrators an improved teaching and learning experience. After costing schools thousands, many of the hyped features of the smartboards have gone unused leaving schools with nothing but an expensive whiteboard. Lack of proper implementation and training are to blame for smartboards being considered one of edtech’s biggest failures.

“I think the industry has some responsibility for this. They should understand that if you’re selling someone something, you have to be upfront and say, “Look, this is complex and difficult, and we’re going to help you understand how sophisticated this is.” — Alan November

Technology should not be seen as a separate entity from learning, technology can and should be integrated into the learning process. Edtech companies that are looking to be apart of any classroom from kindergarten to university level, should aim to properly train teachers and have their technology fully integrated in the learning process, this is key for edtech fulfilling it’s destiny as the next fintech.

The Cautionary Tale of Edtech

In the past two to three decades, the development and accessibility of technology has become a natural part of our everyday lives and with it opened up a whole new range of possibilities in making our lives easier and more efficient. This development is also observed in the educational environment where billions of dollars are being invested each year for purchasing new technology. The lure of it all is quite understandable as it offers a range of (new) possibilities in improving teaching and learning. But while technology in the classroom is here to stay, its use is not always a success.

There are some very interesting edtech products out there but approaching the topic of technology in education is often about finding the balance of selling and buying while also recognising the need.

“Too many school districts buy ed-tech products on the basis of good marketing rather than careful analysis — the way a child is attracted to the hot toy of the Christmas season.” Harold O. Levy

You might have heard about Interactive Whiteboards from late 1990s/early 2000s. This is among the examples that we as an edtech company are all too aware of when approaching educational institutions. Promised as ‘the next big thing’ in educational technology, Interactive Whiteboards were hugely successful when they first came out, but these days they are more of a cautionary tale.

The success of Interactive Whiteboards can probably be accounted to a great sales tactics, rather than exploring whether there is an actually need for the product. Although it promised to be the future of education and innovative teaching, the product ultimately fell flat due to the lack of implementation and the complexity of use. At the end of the day, Interactive Whiteboards were just really fancy and expensive whiteboards.

What can we learn from edtech failures?

Even if you’ve been burned by edtech in the past there is hope. It can be difficult navigating the vast landscape of edtech products and finding the ones that fit best with your school and your classroom. Taking the time to research and having the patience to implement the changes can improve or even revolutionize the classroom. Consider these aspects:

Where is the need? Though a product might look smart, there might not actually be a need for using it. Is the product supposed to solve a perceived need or one that teachers and students actually have?

Are the teachers on board? Oftentimes the lack of implementation or integration of a tool is due to the teachers having received limited or no training. It is well worth the time spent on educating teachers to use the product and let them explore all the possibilities it offers. Equally important is to offer continued support for whatever problems might occur after the introduction phase.

How does it complement the teacher's activities? It should be also noted that in making the decision of buying edtech, it is necessary to consider the ease of use. Teachers already have a range of things to consider and keep in mind; technology should add to simplifying those matters, rather than making them more difficult.

How will the students benefit? It is important to also consider the student perspective when taking a new edtech product into use. Just as for teachers, is the technology amplifying and improving their education or is it just another app in their arsenal of technology.

What is the implementation plan? Implementation takes time and taking on new technology requires teachers (or students) to change the way in which they work and think. It also requires the patience to deal with the challenges of implementing new technology, Rome wasn’t built in a day so don’t expect an entire classroom to be.

Peer Evaluation in Action

Scaffolding for good feedback

The importance of rubrics

Introducing rubrics

When using peer evaluations and peer feedback in a course or class, the evaluation questions are probably the most important thing to focus on for the teacher. This is true both in the case where the evaluations are meant to provide students with additional feedback for learning and in the case where the evaluations are meant as a help for the teacher in giving fair grades.

When searching for guidelines to make a rubric you are likely to find a plethora of websites and articles that want to teach you the basics of how to make a grid and fill it in. However, a rubric is so much more than a well laid out grid. More important than the layout is the description and criteria contained within the rubric. These descriptions are what lead to qualified, valid and reliable assessments of a student’s work.

“In education terminology, rubric means ‘a scoring guide used to evaluate the quality of students' constructed responses’.”
- James Popham.

Rubrics are used to ensure relevant and structured feedback. Good rubrics will guide the feedback-giver, while the feedback-receiver and teacher can anticipate constructive and actionable evaluations. A rubric consists of organized evaluation criteria designated by the teacher. This approach has a number of advantages over open-ended, unstructured feedback design:

As an Assessment Tool

The main purpose of a rubric is it’s ability to assess student’s performance or work. Rubrics can be tailored to each assignment or to the course to better assess the learning objectives. Being able to tailor and customise rubrics means that rubrics can be used for just about any assignment and any course, basically a rubric can be a one size fits all tool.

Learning to correctly construct and use a rubric will result in a time efficient and consistent grading process for both teacher and student. The fact that a rubrics basic purpose is to consistently and fairly assess a student’s work should make it an important enough classroom tool, but there are two more reasons we love rubrics.

As a means to Improve Learning

“This reflective ethos on work produced, fosters communication and the learning cycle to be completed.” — Cox, Morrison, Brathwaite

Using a rubric should be thought of as an active and engaging form of learning. In the ideal situation a student will not read a rubric once, but use the rubric as a way to reflect, analyse and improve their work.

When rubrics are used by both student and teacher alike a rubric creates what Cox calls a “feed forward mechanism” meaning a rubric should allow the student to reflect on their work and focus on how to improve in the future. It’s not just about student’s using a rubric as a form of feedback but giving them an opportunity to use the feedback.

As a Communication Tool

Rubrics can be seen as a communication tool between student and teacher as it aligns expectations and outlines learning objectives for the assignment between student and teacher. One main benefit of aligning and clearly stating expectations is that it creates transparency in grading; grades will seem less arbitrary if students can see what grades are based upon especially if rubrics are being used in peer grading.

Another added benefit is when rubrics define clear learning objectives teachers can quickly and effectively monitor students progress. This allows for monitoring of students that are falling behind, but it is also a great way to adjust course materials and assignments. Rubrics can point to which questions or learning objectives that the whole class is struggling with or what may even be too easy.

General considerations for good rubrics

A rubric consists of a number of questions organized in sections. Each question is complimented by a predefined answer type with specific possible answers (quantitative) or text-based feedback (qualitative). The two answer types can also be combined, by using mandatory comments with quantitative questions.

Questions

Specific

The best feedback is specific, so lead by example and give students specifics about the criteria and the learning expectations for the assignment.

Reinforcement

Rubrics should be assessing if students have reached their learning goals. Therefore, when creating a rubric ask questions that reinforce what has been taught in the classroom and what should be found in the assignment.

Answers

Consistency

Keep the scales the same or similar throughout the rubric, even better keep the scales the same throughout the course. For example, keep to a 3 point scale, using below, meets or exceeds expectations. Do not forget to explain what these expectations are though. Not only will this help students learn how to use a rubric, it also leads to saving valuable time.

Clear

Keep in mind the audience for your rubric. If this is a rubric for your students the language used should be be clear and student-friendly. This includes keeping evaluation scales to under five categories as to not overwhelm students.

Descriptive

Students should be matching a description to the work rather than having to make a judgement purely on their own. Students will trust each other’s feedback more if they feel it has not been solely based on the discretion of their peers but guided by the teachers criteria. This will also lead to transparency in the grading process.

Weighting

The primary goal of a rubric is to evaluate the performance of students. Not all evaluation criteria necessarily carry the same weight in . Let’s consider to simple yes/no questions from an assignment:

Q1: “Was the calculations done correctly?”

Q2: “Was the layout done according to course guidelines?”

Typically, a teacher would consider Q1 as a constaint for performing an excellent assignment, while Q2 is more of a free variable. In this case the weight distribution between the two questions should not be even.

It is recommended to consider importance of criteria prior to formulating the specific questions.

Making effective rubrics

Use short evaluation scales

People are generally not very good at evaluating things on a large evaluation scale. Since it can be hard to differentiate between something that is “very unlikely” and “quite unlikely”, this leads to disagreeing results in the evaluations. In some cases you might want to have 5+ possible answers, but then it is important to make sure that it is feasible for the students to differentiate between the possible answers.

Use examples

When asking students to evaluate the quality of something, it is essential to make sure that they know what defines good and bad. The easiest way to do this is to include specific criteria and examples in the evaluation question. This helps the students get an idea about what constitutes a good solution and provide useful feedback.

Make responses match the question

A great way to guide students in giving proper feedback and evaluations is to provide clear question responses that match the question. Generic evaluation criteria can often be misleading - even when it is hard to imagine how they could be.

Ask about one thing at a time

Often it can be luring to combine multiple questions into a single question to reduce the number of evaluation questions that students have to fill out. Unfortunately this often leads to questions that are harder to answer, and consequently will actually take longer for students decide on. It can also lead to misunderstandings.

Use meaningful sections to break up questions

When you have a lot of evaluation questions, it can often be a good idea to break the questions into meaningful sections. If your assignments require students to solve multiple exercises, it might make sense to have a section for each exercise. It can also make sense to have a section for things that are related to formalia and a concluding section which asks general questions about the work overall.

Having sections in the evaluation questions makes the process of filling them out easier on the students since they can track their progress better. It also helps students in knowing what the relevant context is for the specific questions is. Starting by formulating section labels might very well make it easier for you to subsequently formulate questions.

Combine qualitative and quantitative questions

Generally evaluation questions can be divided into two categories: qualitative and quantitative.Qualitative questions are usually require a text answer and will ask students to provide explanations, arguments and examples. Quantitative questions will often ask students to evaluate something on a scale (either yes/no or a larger scale). Both question types have their strengths and it is often a good idea to combine them.

Qualitative questions requires the students to put their thoughts into constructive written arguments — which is a great way of training the skill of providing constructive feedback. Most times, it is also when receiving qualitative feedback that students get a chance to really learn something. It is hard to use qualitative questions for evaluating performance.

Quantitative questions requires students to assess the quality of something on a scale provided by the teacher. This helps students understand the evaluation criteria and learning requirements for the course. It is for the most part easier for students to fill out quantitative questions, and consequently you can have more of them and force students to look at many different aspects of the work.

By including both qualitative and quantitative questions in a rubric, students need to be both “on the point” and give more constructive and detailed feedback.

Feedback in Action

Download PDF for easy access and sharing.
 

 

Validity in the numbers

Steps to increase validity

Can you trust peer feedback? The short answer is yes, according to research, students are able to give very good evaluations that are often as good or better than their teachers. The longer answer is that for peer grading and peer assessment to be trusted certain steps need to be taken to help ensure its validity.

Anonymous feedback

An online platform, like Peergrade, where both the giver and receiver of feedback are anonymous will offer more objective feedback with a higher quality. This will remove the possible bias of social relations from the process of giving feedback, as student’s might worry about giving critical feedback to their friends.

Additionally, in face-to-face peer review, students are often more concerned about social dynamics, such as whether they will be perceived as mean or embarrass their peers, than providing accurate peer feedback (Christianakis, 2010; Peterson, 2003).

Feedback from 3+ students

In order to assure trustworthiness, students should receive feedback from several of their peers. We encourage teachers using Peergrade to have students peer grade at least three of their fellow students. This is done in order to offer a more objective review of the assignment (and provide a statistical trend). It is also beneficial to the student, the more feedback they receive the more likely the are to receive the feedback that they really need.

“The reliability of several evaluators’ combined ratings is higher than the reliability of a single evaluator’s ratings (Rosenthal & Rosnow, 1991), and this multiple-ratings factor may overcome differences in the reliability of instructors versus students.” (Cho, K., Schunn, C. D., & Wilson, R. W. 2006).

Best results through scaffolding

It cannot be stressed enough that in order to ensure that students are able to give good feedback, it is essential that the teacher has given clear guidelines and prepared well thought out evaluation criteria. Giving feedback needs to be considered a new skill that students are learning, take this into consideration when developing evaluation criteria, as students may require more detailed instruction in the beginning to grow these skills.

Research has namely shown that students are able to give great feedback, but only if they have the guidelines to do so.

Studies at the college level have demonstrated that when students are guided by a clear rubric and held accountable for the quality of their peer feedback, their assessments of their peers’ writing have strong reliability and validity (Cho, K., Schunn, C. D., & Wilson, R. W. 2006).

Accountability

In combination with clear and detailed rubrics, holding students accountable and offering incentives will help ensure validity. When students know that the feedback they write has value and potential consequences, they will be more likely to put the time and effort into creating meaningful feedback. Students will also pay more attention to the rubrics and guidelines provided by the teacher, not only helping them give focused feedback but also encouraging self-reflection and self-assessment.

Feedback on feedback

Feedback cannot always be fair and accurate. The ability to object to feedback is important. 

The insights that support feedback*

Insights, generated through statistics can support feedback.

Level of agreement

Constructiveness

Valued (liked) feedback

Number of flags

Use in grading*

 The outcomes of peer evaluation can be used as components in the grading of assignment. There are different ways to consider peer assessment in relation to grading. In this section we wish to convay a few best practices, based on literature and seeing many examples of real world usage. Generally, the link between , should be based on both best possible evaluation of student performance and as a motivating factor for the students to put an effort into peer assessment.

Assignment performance

Feedback performance

Weighting of questions

Limitation. Don't directly translate a peer evaluation score to grades.

Validity in Action

Peer grading can provide valuable feedback for students while also improving critical thinking skills and encouraging self-reflection. Those benefits alone are worth implementing peer grading in the classroom, but when done correctly students can also produce valid and trustworthy grades.

Getting started

Typical use cases

Getting started

Eager to introduce peer evaluation in your lectures? This approach will lead to smooth sailing.

  1. Define the purpose and use of peer evaluation in your course
  2. Find the tools to support peer evaluation. Peergrade handles all aspects and integrates with popular Learning Management Systems (full disclosure: Peergrade is offered by Masterclass co-authors). Product reviews are outside the scope of this class, so we encourage you to look around for other tools that could satisfy your needs. Start by consulting your institution and see if they currently license Peer Evaluation software
  3. Test of the tools, to establish your first take on best practices. Discuss best practices with collegues that use the same - or similar - tools.
  4. Integrate the use of peer evaluation into your course material
  5. Run your course and make sure students are informed about the purpose as well as practical aspects

Tools for an effective peer assessment system

It is important to have a systematic approach for handling peer assessment. The approach can be very manual or very automated. The latter is typically preferred, as it does not rely on the teacher to facilitate the process. Let us have a look at the common requirements for a good peer assessment, before reviewing selected tools that can help you achieve an effective system.

The components of a good peer assessment system

The primary aspects that should be covered are:

  • Hand in assignments
  • Match feedback providers with assignments to assess
  • Rubrics design
  • Anonymity
  • Intuitive for both students and teachers
  • Deadline management & late hand-in options

Secondary aspects that are typically very valuable:

  • Structured approach to feedback
  • Group hand-in management
  • Categories
  • Flags and comments for feedback
  • Hand-in performance overview
  • Feedback quality measure
  • Feedback validity measure
  • Plagiarism check
  • Teacher & self assessment
  • Easy overview of results

Tools

These two tables provide an overview of features in selected tools. Each tool is briefly introduced below.

  A B C      
Blackboard            
Peergrade            
             

 

Blackboard is a Learning Management System (LMS) ...

Peergrade is made specifically for peer assesments ...

Literature Review & References

Peer Evaluation

...

Validity

Cho, K., Schunn, C. D., & Wilson, R. W. (2006). Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology, 98(4), 891–901. doi:10.1037/0022–0663.98.4.891

Nicol, D., Thomson, A., & Breslin, C. (2013). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122.

Schunn, C., Godley, A., & DeMartino, S. (2016, July/August). The Reliability and Validity of Peer Review of Writing in High School AP English Classes. Journal of Adolescence & Adult Literacy, 60(1), 13–23

Rubrics

Susan M. Brookhart (2013). How to Create and Use Rubrics for Formative Assessment and Grading (book)

G.C. Cox, J. Morrison, B.H. Brathwaite (2015). The Rubric: An Assessment Tool to Guide Students and Markers (article)

Feedback

Providing feedback for student learning; Using eVALUate to improve student learning; Curtin University of Technology; Studies in Higher Educationhttps://evaluate.curtin.edu.au/local/docs/5providing-feedback-for-student-learning.pdf

Giving Feedback; Keeping Team Member Performance High, and Well-Integrated; Mind Tools; https://www.mindtools.com/pages/article/newTMM_98.htm

Strategies to enhance peer feedback, Assessment for Learning; http://www.assessmentforlearning.edu.au/professional_learning/peer_feedback/peer_strategies_enhance.html

Formative assessment and selfregulated learning: a model and seven principles of good feedback practice Vol. 31, No. 2, April 2006, pp. 199–218; David J. Nicola and Debra Macfarlane-Dick; http://web.iaincirebon.ac.id/ebook/Indrya/Assesmen/Formative%20assessment%20and%20self-regulated%20learning.pdf

Assessment for Learning, Sally Brown, Learning and Teaching in Higher Education, Issue 1, 2004–05; http://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/FilestoreDONOTDELETE/Filetoupload,120807,en.pdf

Giving better feedback for better performance. Go beyond ‘good job!’ to help people learn and thrive; Dr. Kristen Swanson, Director of Learning at Slack; June 2016; https://slackhq.com/giving-better-feedback-for-better-performance-3eab4036562b#.odecv8pur

Dyads Versus Groups: Using Different Social Structures in Peer Review to Enhance Online Collaborative Learning Processes by Francesca Pozzi, Andrea Ceregini, Lucia Ferlino, and Donatella Persico in International Review of Research in Open and Distributed Learning. Volume 17, Number 2.

Hattie J. & Timperley H. (2007). The Power of Feedback

Sadler, R. (1989). Formative assessment and the design of instructional systems

Motivating students

What the Best College Teachers Do by Ken Bain

Twenty Tips on Motivating Students, University of Nebraska-Lincoln

Motivating Students, Vanderbilt University, Center for Teaching



FEEDBACK & DISCUSSION

Should recent trends be included in The big picture -> benefits, as is now the case?

  • Yes
  • No