Skip to main content

Simulated Peer Review

Why Use Simulated Peer Review

A primary value of adapted journal article teaching is the emphasis on student-generated analysis, discussion and debate. For a professor to tell students what they should think ahead of time defeats the main point of the exercise. This kind of learning activity provides strong and convenient incentive for professors to extend their traditional "stand-and-deliver" teaching style. And, we believe, teaching in this way will stimulate students to become more intellectually engaged and active in their learning.

While this view is often supported by the facts, professors should ask "How will my students ever learn to think for themselves, if I don't require it?" Of course, many students are not knowledgeable enough, and they will find such lessons difficult. But how else can they learn to do difficult things if they are not challenged? Thinking is a skill that can be learned. The main point of this kind of teaching is to help students learn to think about academic matters, not just memorize answers for exams. In short, knowing the WHAT of a given discipline is not enough. Students also need to know the WHY, HOW, AND SO WHAT.

A major goal of this kind of instruction is to show students how to develop their creative and critical thinking capabilities. Simulating peer review is a logical way to spur students to be more insightful, as well as show how to scrutinize scholarly work before publication. To guide student thinking, we provide in the lesson plan some 24 questions that the students should address in their simulated peer review. In terms of pedagogy, this exercise embraces every level of Bloom's Taxonomy.

Basic Requirements

  • Students (preferably grouped as learning teams) are assigned an ARR and conduct a simulated peer review.
  • Some form of deliverable should be required (written analysis, poster, class presentation, etc.).
  • Student reviews should graded primarily for their critical and creative content
  • Professor should follow up to show how a professional would have evaluated the ARR.

Procedure & Scaffolding

Students should write or give an oral presentation that applies creative and critical thinking to the paper. Poster reviews are another option for a deliverable. Students should conduct the analysis of the paper's primary sections sequentially. The inquiry is probably most effective if scaffolding questions are presented.

Scaffolding Questions.

Note: simply stating an opinion is not sufficient. Where opinions are presented, they must be defended with facts and logic.

Introduction

  1. Was there an explicit hypothesis? If not, what was the implicit hypothesis?
  2. How reasonable does the rationale seem? Why does it seem reasonable or not reasonable?
  3. What are some alternative ideas that were not considered.**
  4. Does this research seem scientifically important? Does it have practical importance? Why or why not?

Methods

  1. Is the design adequate? Why or Why not?
  2. How well do the control groups serve as checks on variables that could influence results other than what is being tested? Why or why not?
  3. Describe the negative control group and its function? Are there important variables that the control group does not account for?
  4. Is there a positive control group? Is one needed? Why or why not?
  5. Is double-blind testing needed and used? Why or why not?
  6. Do the data-collecting approaches or devices seem Appropriate? Are they sensitive enough for what is being tested?
  7. What other approaches or devices might have been used?**

Results

  1. Do the results support the hypothesis or not? How convincing is that support?
  2. Do you notice anything of potential importance in the data that was not commented on by the authors?**
  3. Is the variance in data large enough to suggest that some variables are not being controlled? What might these be?
  4. Apart from the statistical effect, what is the magnitude of the 'treatment' effect? Is it large enough to be of much practical importance?

Discussion

  1. Summarize how the authors discussed the results in terms of their original hypothesis.
  2. Did they point out implications that go beyond the hypothesis?
  3. What implications did the authors perceive that go beyond the original hypothesis.
  4. Do you perceive any other implications? **
  5. What ideas for future research did the authors generate?
  6. What ideas for future research do you generate?**
  7. Note any important information or ideas that were not commented on by the authors and explain the implications.**
  8. Does the author state a 'take-home lesson?
  9. How would you state the take-home lesson?**

**Creative ideas are expected

Team Learning

This kind of exercise is best performed by small student teams (4-6/group), because the intellectual challenge could exceed the capabilities of any one undergraduate. This kind of teaching should have special appeal to the Colleges of Business, Engineering, Medicine, and Veterinary Medicine, where team learning is already an accepted practice.

The formalisms of proper collaborative learning will be explained to professors to assure that the process is optimized. To summarize the basic principles: teams should be heterogeneous, every member of the team must have a defined role (our lesson plan explains the roles for APS), an academic deliverable is required (in this case, simulated peer review), and a significant portion of the grade must be a group grade (grading rubric options are provided).

Working as a group confers other special advantages, which we hope the professor will explain to the students (see "Instructions to Teams" below). Students may have had bad experiences with group work in the past (such as non-contributing group members or poor social dynamics). The professor should explain how this group-work approach will help students, both in terms of academic ability but also in terms of social attitudes and skills. Tell students that they will benefit from this exercise because it:

  1. Encourages them to think for themselves, yet in the context of a team effort.
  2. Authentically reflects how scholarship is usually done these days - by a team of researchers who bring different knowledge and skills to the problem. The old days of a lone scientist locked up in an "Ivory Tower" are mostly gone. Science is now so advanced that no one scientist knows enough or is smart enough to do it all alone. Scientists must learn how to work in teams, often with people who were former strangers, often located in another city or even another country.
  3. Rewards them for thinking "outside the box" and generating good ideas.
  4. Requires them to assert their ideas and thinking, but at the same time to be able to defend them with peers.
  5. Teaches them to disagree without being disagreeable.
  6. Teaches them to accept and benefit from legitimate criticism of peers.
  7. Requires them to contribute to the welfare of the group, rather than to be selfish
  8. Allows them the experience of being on a winning team, one that has learned how to work together, sharing the joy of accomplishing something none of them could have done alone. They can be proud to have been an important part of the group.

How to Promote Team Cohesion

  • Keep group small (4-6 students/group)
  • Assign specific team roles (leader, librarian, editor, presenter, whip, etc.)
  • Group grade
  • Require public display of the deliverable (class presentation, poster, web blog, etc.)

How to Eliminate Malingering

  • Promote individual role functions, team identity, bonding, competing spirit.
  • Have each individual show the professor a draft of answers to all scaffolding questions BEFORE the team deliberation begins.
  • Include some form of individual grading along with the group grade.

Blended Learning

Combining in-class work and out-of-class work on a simulated peer review is one form of "blended learning." Performance of out-of-class work by a learning team is most conveniently done in Internet environments that let team members share the same digital workspace.

A very useful part of the simulated peer review is for a team to learn about various Web 2.0 environments and decide which one they would like to do their work in. Perhaps their very first task is to assign Web 2.0 options and have team members report back to the team how the software works and the pros and cons of its use in simulated peer review. The advantages of making such a decision include:

  • Team members have an "ice-breaking" activity that helps them get to know each other and form working relationships
  • They learn about available technologies, some of which they will probably wish to use later for other purposes.
  • The team has to think about what information and ideas it needs to collect, as related to the scaffolding questions, and which environment would be the most "user-friendly" and Appropriate for that purpose.

Most of these Web 2.0 systems operate in "the cloud," that is the software and data remain on a Web server, accessible at any time from any computer. We will provide guidance about the useful software collaborative environments through e-mail, Web documents, and seminars.

Grading of Simulated Peer Review

To be fair, grading team performance requires combining team and individual grades. One approach is to give a grade for each team, with all members getting the same grade, and averaging that in some weighted fashion with an individual grade derived from scores from team members, content test scores or other related work.

A suggested scheme for integrating team and individual grades is derived from that used by others ( http://sciencecases.lib.buffalo.edu/cs/teaching/assessment/student-case-work.asp). The process uses anonymous peer evaluations on a form teams fill out at the end of the semester. Students give each team member a score based on their contributions to group projects throughout the course. To stop a student from being too generous to friends, a limit has to be placed. Multiple options exist. A professor can insist that every score has to differ by at least 5% (or 10% or whatever). That way, top performers will always get recognized, as will bottom performers. Then, individuals can be given a proportional amount of the group grade as their individual grade.

It may be advisable to give a practice peer evaluation about one-third or one-half of the way through the semester, so that each team member can made formative adjustments in their work and interaction with fellow team members. Low-scoring students are advised to fix things, perhaps by talking to the group and asking how to compensate for their perceived weakness. The professor can always intervene to helping solve group-dynamic problems.

Grading Rubric

The rubric below can be used for presentations that are written, oral, or poster. Items and scoring are easily modified. We chose to put emphasis on items that require critical and creative thinking.


Questions

Max. Points

Bonus Points*

What is the problem the research is trying to address?

5

 

What is the main question being asked about the problem?

5

 

Are there other relevant questions that could be asked?

10

 

How did the authors frame the question to make it a testable hypothesis?

10

 

Summarize the methods used to test the hypothesis.

10

 

What other methods could be appropriate?

10

1-5

Identify the dependent and independent variables.

10

 

What was the control? Was it adequate? Why or why not?

10

1-5

Summarize the results obtained.

5

 

In what way did the results confirm (or refute) the hypothesis?

5

 

What ideas did the results generate for the authors?

5

 

What ideas did the results generate for you (not stated by authors)?

15

1-5

*Awarded only for exceptional insights.

References

  • Alberts, Bruce. (2009). Redefining science education. Science, 323: 437
  • Raphael, Ellen. (2009). Teaching peer review. The Scientist, May. p. 24