Feasibility Study of Using Calibrated Peer Review in the Large General Chemistry Laboratory Class
Ann Biomed Eng. 2018; 46(three): 513–524.
Using a Education Intervention and Calibrated Peer Review™ Diagnostics to Improve Visual Advice Skills
Ann Saterbak
1Section of Biomedical Engineering, Duke University, Durham, NC USA
Anoosha Moturu
2Baylor College of Medicine, Houston, TX United states of america
Tracy Volz
iiiProgram in Writing and Communication, Rice Academy, 6100 Main St – MS 630, Houston, TX 77005 U.s.a.
Received 2017 Jun 30; Accepted 2017 Oct 17.
Abstruse
Rice University'south bioengineering section incorporates written, oral, and visual communication pedagogy into its undergraduate curriculum to aid student learning and to set up students to communicate their knowledge and discoveries precisely and persuasively. In a tissue culture lab course, we used a self- and peer-review tool called Calibrated Peer Review™ (CPR) to diagnose student learning gaps in visual communication skills on a affiche assignment. We then designed an active learning intervention that required students to practice the visual communication skills that needed improvement and used CPR to mensurate the changes. Later on the intervention, we observed that students performed significantly better in their ability to develop high quality graphs and tables that stand for experimental data. Based on these outcomes, we conclude that guided chore do, collaborative learning, and calibrated peer review tin can exist used to improve engineering students' visual communication skills.
Keywords: Tissue civilisation, Data analysis, Information interpretation, Peer review, Assessment, Graphs
Introduction
Background on Visual Communication in Engineering science
Importance of Visual Communication in Engineering
In The Engineer of 2020: Visions of Engineering in the New Century the National Academy of Engineering identifies advice skills as a top priority for engineering graduates to be successful.18 In addition to written and oral skills, engineers need to possess visual communication competencies such as sketching, modeling, rendering, and data presentation.4,9 To develop these competencies, many engineering faculty assign reports, presentations, and posters that characteristic images, computer-aided design drawings, figures, and tables, yet it is not clear students receive much pedagogy on how to create constructive visuals.
Some applied science faculty have found that students are sick-prepared to visualize and translate information and have taken the initiative to explicitly teach students at every level how to become more practiced in these areas.6 Taylor and Pulford, for example, note that prior to enrolling in capstone design, engineering students receive little instruction in design principles or core skills such as observation and envisioning, which aid in the preparation and interpretation of visuals.25 Other engineering faculty accept attempted to address this trouble earlier in the technology curriculum. Sherwood and Atabile were dissatisfied with the quality of the figures and tables created by juniors in a mechanical engineering design course and implemented a rubric to signal their expectations and to evaluate students' work.24 Dupen introduces outset-twelvemonth students to information presentation in lab reports through a combination of mini-lecture, consignment-specific instructions, and repeated practise.8 A scaffolded approach to learning that involves frequent exercise and feedback is needed to achieve a high level of visual communication competency.
Prior Work and Intervention to Target Form Learning Outcomes
Recognizing the importance of multi-modal advice, bioengineering faculty at Rice University partnered with professional communications faculty to develop and implement a vertically integrated, advice-enhanced curriculum.21,23 As part of this curriculum, the authors created an assignment in BIOE 342, a tissue culture laboratory form, in which students develop technical posters summarizing their experimental results.26 This consignment is specifically designed to develop students' skills in data presentation and analysis of results. Nosotros integrated a tool called Calibrated Peer Review™ (CPR) with the poster assignment, which immune the states to capture students' evaluations of the poster drafts. The CPR information combined with instructor evaluations made it possible to conduct a fine-grained analysis of the posters' strengths and weaknesses.
In 2007 we launched a enquiry project to investigate the level of agreement—or lack thereof—betwixt the instructor'due south evaluations of students' posters and students' evaluations of their own and their peers' posters.26 Information nerveless via CPR betwixt 2007 and 2009 indicated that students were struggling to construct technical arguments and to create effective visual displays of their results. The CPR data confirmed our perception that students were skilled at evaluating their peers' posters on lower-club cerebral tasks such equally the formatting of figures and tables and elementary reporting, but they performed poorly on college-order cognitive tasks such equally analysis, synthesis, and evaluation. More than specifically, students struggled with tasks such as condensing and presenting graphical data, correctly interpreting graphical representations of data, and synthesizing and stating key results, peculiarly equally they chronicle to linking experimental variables with measured cellular furnishings.22,26
Therefore, in 2010, we initiated a pedagogical intervention to target two learning outcomes (LOs): (1) Develop loftier quality graphs and tables that represent experimental information, and (2) Correctly interpret and conspicuously summarize the experimental results. This paper describes CPR, the tissue civilisation laboratory, and the classroom activities directed at the 2 LOs. The newspaper as well discusses the results of the intervention on pupil performance.
Groundwork on Calibrated Peer Review™ (CPR)
CPR facilitates asynchronous, bearding electronic peer review.26 It was designed to facilitate peer-graded writing assignments in large lecture courses and has been widely adopted by educators in a variety of disciplines including biological science, chemistry, technology, and medical education.1,12,14,15 Faculty have used CPR to appraise essay writing,28 lab reports and design reports,12 technical writing,16 and course content.17 Nosotros were early on pioneers in the use of CPR to teach technical posters.22,26 Recent improvements to CPR 6 include a new interface specifically designed to support peer review of graphics and multi-media files.five,six,27
To complete the CPR assignment, students upload their affiche drafts in the Text Entry stage. In the Calibration phase, students apply evaluation criteria to three sample posters, which take been selected and rated by the instructor. Students' ratings of each sample affiche must fall within a prescribed standard deviation established past the instructor before students can enter the Review stage. During the Review stage, students use the aforementioned criteria to evaluate three peers' texts. Afterwards reviewing their peers' texts, students complete the Self-Assessment stage. When the assignment concludes, students can view their results.
Instead of using CPR to supplant teacher grading with peer grading, we use CPR in BIOE 342 to give students more feel with peer review and as a mechanism for determinative cess. Students receive credit for their critiques in CPR, but their poster grades are adamant by the instructor.
Educational Framework
Tissue Culture Laboratory
The Tissue Culture Laboratory (BIOE 342) is i of the required laboratory courses in the bioengineering curriculum that includes a significant communication component. The teacher established half-dozen course goals: (1) Demonstrate proficiency with tissue culture techniques, (2) Keep a laboratory notebook, (iii) Perform assays to narrate prison cell viability and part, (iv) Clear links between changes in cellular environmental conditions and cellular office, (v) Improve communication skills, and (6) Refine critiquing skills. To achieve the first four goals, students learned sterile technique and performed viability, attachment, and proliferation assays using a fibroblast cell line. The experiments produced quantitative information such every bit cell number, jail cell density, or fractions. To run into the final two goals, students prepared a affiche of their laboratory results and afterwards critiqued their peers' posters. This 5-week, junior-level grade carried 1 h of credit and enrolled approximately thirteen students per department, with four sections offered per twelvemonth. Students spent 6–x h in lab each week and attended six 50-min lectures.
Standard Poster Consignment
This section describes the standard poster assignment prior to the intervention. In the context of this course, "affiche" refers to x–12 slides that nowadays experimental objectives, methods, results, and conclusions. This format has 2 advantages over conventional conference-mode posters. Outset, it focuses students' attention on constructing a technical argument without having to tackle macro-level layout and pattern issues, which are addressed in other bioengineering courses. 2d, it allows students to avert printing costs. Students in the course do non present their posters orally at a poster session.
To prepare students, the bioengineering teacher introduced the assignment and expectations. Then, the communications teacher lectured on the poster genre, technical argument, and data presentation. Following this lecture, students met in pocket-size groups for v–10 min to critique 3 poorly designed slides produced by one-time BIOE 342 students and reported their observations to the class. During standard instruction (2007–2009), these activities occurred in lecture period 5 (Tablei).
Table i
Comparison of class schedule before and after the intervention.
The intervention schedule removed one lecture period on tissue civilisation experimental details. In its identify, an active learning lesson intended to increment proficiencies in LO1 and LO2 was added.
Later, students submitted affiche drafts, which were graded past the teacher and evaluated by their peers using CPR. During the Scale phase, students reviewed three sample posters based on a periodical article that had similar tissue culture experiments to the ones the students had completed.13
To critique the posters in the Scale and Review stages, students used 14 evaluation statements related to presenting data in figures and tables, interpreting data in words, and applying disciplinary conventions (Tabletwo). Based on the feedback students received from the instructor and their peers, they revised their posters for a final grade. The grade form was weighted heavily toward this assignment: poster draft (ten%), CPR performance (10%), and revised poster (20%).
Table 2
Evaluation statements.
Students and teacher used the statement number (Stmt), evaluation argument, and rating calibration to critique the posters. A/B/C scale: A loftier/strongly agree, B moderate/neutral, C low/strongly disagree. Yes/No scale: Yes agree, No disagree. 1–x scale: 1 is low, 10 is loftier.
Intervention Poster Assignment
To address specific areas of students' underperformance in the technical poster assignment identified earlier,24 we implemented a pedagogical intervention in spring 2010. This type of intervention is based on job-based exercise and cooperative learning and has been shown to ameliorate student performance in a laboratory setting.two,20 The class outline, experiments, assignments, and relative weighting of the assignments remained the same during the intervention period (2010–2012).
The same fourteen evaluation statements were used during the intervention period (Tabletwo). Statements i–iii were linked to LO1: Develop high quality graphs and tables that represent experimental information, and Statements 4–8 referred to LO2: Correctly interpret and conspicuously summarize the experimental results. The other statements were not addressed by the intervention. To meet these LOs, only one curricular change was made, namely the replacement of the fourth lecture on tissue culture experimental details with a flow of explicit instruction and practice toward the two LOs (Tabular array1).
The 50-min intervention included two active learning activities. To address LO1, each educatee submitted one draft results slide. The instructor grouped submissions into triads featuring figures/tables of variable quality. During class, students gave each other feedback in anticipation of revising the slides for their posters. The instructor also projected some sample slides for group discussion. Students did not use a rubric for this activity.
To address LO2, students were given figures and tables based on data derived from the tissue culture experiments. They analyzed the materials in small groups and wrote informative slide titles or stated the key results in brusque, specific phrases.
Methods
Overview of Research Questions
To evaluate the effect of the intervention of additional specialized instruction and do in preparing figures and tables and writing about key results, data from the posters was systematically nerveless and analyzed. At the outset of the intervention, we hypothesized that peer-evaluation and self-evaluation would increase in accuracy relative to the teacher's evaluation, and that overall pupil performance would increment. Nosotros posed three research questions (RQs) that compared the standard and intervention conditions:
RQ1.
Did the ability to accurately peer-review improve as a issue of the teaching intervention?
RQ2.
Did the ability to accurately self-evaluate improve as a result of the teaching intervention?
RQ3.
Did pupil functioning increase as a result of the education intervention?
Data Collection
BIOE 342 posters were nerveless for this report over a six-year catamenia, from 2007 to 2012. The information for each pupil'due south affiche included the following items: iii peer-evaluations of the draft, one cocky-evaluation of the draft, 1 instructor evaluation of the draft, 1 instructor evaluation of the terminal. Peer- and self-evaluations, consisting of xiv statements, were gathered through CPR (Tableii). The teacher manually graded the draft and last posters using the aforementioned fourteen statements. If the data for a poster was incomplete (e.g., missing a peer-evaluation), the poster was removed from the data set. Standard information was nerveless from 2007 to 2009 and yielded a full of 105 posters. Intervention data was collected from 2010 to 2012 and yielded 103 entries. All data was collected and used in compliance with university IRB requirements.
Information Analysis for Statements i–13 (Alphabetic, Categorical Data) for RQ1 and RQ2
For Statements ane–thirteen, the responses were alphabetic, categorical data (e.g., A/B/C or Yes/No). Since each student submitted a unique poster based on his or her data, the posters could non be compared to a reference or standard poster. Thus, the instructor evaluation of the affiche was established every bit the standard (since the teacher was the but abiding gene beyond the multiple semesters of educatee submissions). Thus, the raw data for Statements one–13 was reported as the difference between teacher and peer-evaluation or as the difference between the instructor and self-evaluation. To address RQ1, an analysis of peer-evaluation relative to instructor evaluation (PvI) was completed. To accost RQ2, an analysis of cocky-evaluation relative to instructor evaluation (SvI) was completed.
Pace 1: Create Bins as a Means to Compare PvI and SvI
Because the data was alphabetic and categorical (A/B/C or Yes/No), we characterized the difference between the instructor (the standard) and the peer (or self) scores using the terms "college than," "lower than," and "agree." Comparison the data was also complicated by the apply of two different scales within CPR, a 3-signal scale (A/B/C) and a 2-bespeak scale (Yep/No). Since the departure of the score given by the peer or self from the score given by the instructor could vary in magnitude, we adopted the terms "by 1" and "by ii" to stand for the difference betwixt the response and the standard. For statements using the three-point scale (A/B/C) and 2-point calibration (Aye/No), Tabular array3 captures the possible ratings and how they were categorized.
Tabular array 3
All possible 2-betoken and iii-point calibration differences.
The bin is determined by the difference between the teacher and the peer or self. Given a Argument, the scores are divers: Yep as agree, No every bit disagree, A as loftier or strongly agree, B as moderate or neutral, C as low or strongly disagree. Peer or cocky scores are compared to the instructor'south scores. There are 3 bins on the ii-bespeak scale, and five bins on the 3-bespeak scale.
The bin was defined every bit the departure of the peer (or self) relative to the instructor. Step 1 was executed four times for each affiche, once for each of the 3 peer responses and in one case for the self-evaluation. To this stop, rated data (A/B/C, Yes/No) from Statements 1–thirteen was transformed into binned information.
Step 2: Aggregate Bins across Each Statement
The second footstep in the data assay was to sum the numbers within each bin to become a distribution for each statement. Information was summed inside a handling (i.e., standard or intervention) for the PvI or SvI. The output of the summations was a contingency table and a histogram for each statement that reported the number of responses as a office of bin (east.one thousand., Fig.1 represents PvI for Statement ane).
Comparing of the standard vs. intervention approaches for peer-evaluation of Statement 1 relative to the instructor. The number of responses is shown equally a office of the departure in score. The bars bear witness the number of times that the peer-evaluation was lower by 2, lower by 1, agrees with, higher by 1, or higher past 2, as compared to the instructor'due south evaluation.
Step 3: Compare Standard vs. Intervention Conditions Using Chi Squared Exam
Contingency tables presented the changes across handling for peer vs. teacher (i.e., PvI) and for self vs. instructor (i.east., SvI). Contingency tables were evaluated with a χ ii test to compare the standard and the intervention data under the zero hypothesis that both sets came from the aforementioned population. The χ 2 test (alpha set to 0.05) was chosen for this alphabetic, categorical data considering information technology uses the standard condition data to create expected values for each ordinal category in the intervention condition data.
The P value was calculated based on the χ 2 distribution and the degrees of freedom. A few of the contingency tables expressed random zeros. When this occurred, a small constant of 0.00001 was added to every prison cell in the contingency table.three
Stride 4: Examination for Convergence
A exam for convergence was completed only if the P value was <0.05. The desired modify was a convergence of responses to the "Agree" bin, as this showed an improvement in a educatee's ability to match the instructor's evaluation ratings. In society to bear witness convergence subsequently intervention, the "Agree" bin must have contained a higher number of responses, and the other bins must have shown a lower number of responses.
Data Analysis for Argument 14 (Continuous, Numerical Data) for RQ1 and RQ2
Statement 14 evaluated the overall quality of the poster. Peer- and self-evaluation data was collected through CPR on a 1–ten scale. Overall poster quality grades were given by the instructor on a 100-point scale (range 60–96). These grades were linearly scaled downwards to a 1–10 scale using the formula: scaled score = (instructor grade %—lx%)/4 + ane. For instance, A (96%) = x, A − (92%) = nine, …, D + (64%) = 2, D (sixty%) = 1. For Statement 14, the responses were causeless to be continuous.
For the PvI comparison, the mean of 3 peer-evaluation scores for Statement 14 was calculated and graphed against the teacher score in scatterplots. Because data was continuous, averages (rather than differences) were used to present values. For both the standard and intervention conditions, linear regression equations and r values were computed. Convergence to instructor scores is shown by a motion towards a slope of 1 in the regression equation and an increase in the r value (i.eastward., closer to 1) from the standard to the intervention condition. For the SvI comparison, the analysis of Statement 14 was almost identical to the PvI comparison, except no averages were taken, since there was only ane self-evaluation per poster.
Data Assay for Statements 1–13 for Instructor Scores for RQ3
To address RQ3, an assay of the teacher's scores before and after the intervention was conducted. Specifically, this analysis examined raw instructor scores for Statements 1–13 during the standard and intervention weather condition. The four-footstep process outlined in the "Data Analysis for Statements 1–13 (Alphabetic, Chiselled Data) for RQ1 and RQ2" section was used.
To check the direction of this shift, the "mean" of the scores was calculated for the standard and intervention conditions. The A/B/C responses were coded as 3/two/1, respectively and the Y/Northward responses to 2/1, respectively. Averages of these scores for each statement were then calculated and compared between conditions. A higher boilerplate in the intervention condition confirmed that the score for this argument improved following the intervention.
Results
Throughout the Results section, Statement 1, "Key results are presented clearly in graphs and tables," is used as the exemplar effigy for histogram data. Tabular array4 summarizes the data for all RQs as they chronicle to LO1 and LO2 and Statements 1–13.
Table 4
Results of statistical analysis for statements (Stmt) ane–13 organized by learning outcome (LO).
Results bear witness P value comparison of standard vs. intervention approaches for research question 1 (RQ1) (peer-evaluation of posters relative to the instructor), RQ2 (self-evaluation of posters relative to the instructor), and RQ3 (educatee performance, as assessed by the instructor). Statistical significance, reported as a P value, was evaluated with a χ 2 exam. Convergence (i.e., student resembled teacher) and improved functioning (i.e., performance shifted toward a higher score) was tested when P < 0.05. Convergence or comeback is marked every bit Aye or No, with blank as Due north/A (i.e., P value non significant). For like shooting fish in a barrel visual reference, blocks are shaded when there is a statistically significant difference and convergence (RQ1 and RQ2) or comeback in performance (RQ3).
Effects of Pedagogy Intervention on Students' Ability to Peer Review (RQ1)
Effigyone shows how peer-evaluation was lower than, in understanding with, or higher than the instructor's evaluation for Statement i PvI. Using the standard teaching method, students overrated peers in 165 of 315 times (52%). In 128 of 315 times (41%), the peer and instructor assigned the same score. In 22 of 315 times (7%), students underrated their peers relative to the instructor.
After the intervention, there was a shift in the distribution of peer-evaluation equally compared to instructor evaluation. Overall, the percentage of students who overrated their peers equally compared to the instructor decreased from 52% to 36% (110 of 309 times). The number and percentages of students who agreed with the instructor increased from 40% to 55% (169 of 309 times). Later the intervention, 10% of the peers' ratings for Argument 1 were more than critical than the instructor's.
As noted in Tableiv, the P value for Statement 1 was 0.000084. This difference converged, significant there was a shift toward the center. This indicates a statistically significant change in distribution towards greater agreement with the instructor later the intervention.
Contingency tables and related information analysis were completed for Statements 1–xiii. As shown in Table4 in the RQ1: Peer vs. Instructor column, statistically significant changes were seen for LO1 (Statements 1, 3), LO2 (Statement 8) and command (Statements 12–13). For Statements i, 3, and 8, statistically significant differences were shown to converge, meaning that evaluation of their peers' work aligned more closely with the instructor's evaluation afterwards the intervention. For Statements 12–xiii (controls), convergence was not seen.
LO Overall (Argument 14) required a holistic score of the overall quality of the poster, "How would you rate this text/affiche?" Figure2a is a scatterplot of the boilerplate peer score vs. the teacher score for the standard condition. For the standard condition, there was a weak correlation (r = 0.51) betwixt the average peer score and the teacher score. As shown in Fig.iib, the correlation improved to r = 0.68 for the intervention condition. The shift toward better correlations shows students align amend with instructor on the holistic form of the poster. Overall, peers mostly gave higher scores than the instructor, especially for the posters that the instructor scored the lowest.
Scatterplots of Statement 14, the overall score of the poster (on a ane–ten calibration), for the standard (a) and the intervention (b). For each poster, iii peer scores are averaged to one value, which is plotted against the instructor score. The size and color of the dot represent the number of responses at that location.
Effects of Instruction Intervention on Students' Ability to Self-Evaluate (RQ2)
Contingency tables and histograms to compare students' ability to evaluate their own work against the standard of the instructor were created for Statements 1–thirteen using a similar method to RQ1. The information analysis from these graphs is summarized in Table4 nether the RQ2: Self vs. Instructor column. Statistically significant changes subsequently intervention were seen in all statements for LO1 (Statements 1–3). All of these changes were shown to converge, which means for these three statements, a educatee's evaluation of his or her ain work more closely matched the instructor's evaluation after the intervention. There was no divergence or convergence for statements related to LO2 or command.
To evaluate Statement 14, we generated plots like to Fig.two for cocky-evaluation score vs. instructor score (data not shown). For the standard condition, there was no correlation between self- and teacher evaluation (r = 0.18). After the intervention, the r value rose to 0.37, suggesting a weak correlation. Again, students generally gave higher scores to their own work than the teacher did.
Furnishings of Teaching Intervention on Students' Ability to Nowadays Key Results (RQ3)
Figurethree presents CPR scores given past the instructor to the students' posters on Statement 1 before and after the intervention. During the standard administration, 17% of the students scored an A (high/strongly concord), 49% of the students scored a B (medium/neutral), and 34% scored a C (low/strongly disagree). After the intervention, the scores shifted dramatically. Forty percent scored an A, 51% scored a B, and only 9% scored a C. Thus, the instructor's ratings indicate that students substantially improved in their power to create graphs and tables that clearly present the key results.
Instructor scores before and after intervention for Statement 1. A score of A indicates high agreement with the statement; a score of B indicates moderate understanding, and a score of C indicates depression understanding.
As shown in Tablefour under RQ3: Student Operation, statistically significant changes were axiomatic in LO1 (Statements 1–three) and LO2 (Statements half-dozen, 8). For all 5 statements, educatee performance improved. There was no difference or shift in direction for statements related to the control.
Discussion
Using CPR™ to Assess Pedagogical Interventions
We successfully leveraged CPR's functionality in a novel way to place and address gaps in student learning related to the disquisitional analysis of experimental data and technical communication. In our previous work,22,26 nosotros analyzed the peer review and self-evaluation information nerveless in CPR. This analysis revealed practiced alignment for PvI and SvI ratings on statements requiring comprehension and application of knowledge, likewise every bit on the argument asking for an overall rating of a poster'southward quality.26 In order to meliorate the quality of the grade and student functioning, we designed educational interventions to target poor alignment related to higher-society thinking tasks, specifically in information presentation and interpretation.
Importantly, we used CPR in the revised course design to measure changes in students' performance. We found that subsequently guided exercise on figures and tables, students showed better alignment with the instructor on some statements and that their skills improved in particular areas. Ultimately, CPR provided a powerful platform that enabled us to hone in on specific weaknesses in students' visual communication skills and to determine whether a new pedagogical approach yielded better outcomes. With its automated, online system, CPR can be a helpful tool for engineering educators equally they seek to deliver customized education to their students.
Mixed Intervention Improved Students' Visual Advice Skills
In a laboratory setting, a combination of interventions can help students develop their "scientific literacy," which includes the student'southward power to collect data, clarify results, and understand cause and consequence in an experiment.20 In a pedagogical intervention similar to ours, Carter showed that explicit instruction and practice using LabWrite helped students with many aspects of lab reports, including representing and interpreting data.7 In another example, Taylor and Builford adopted a critique approach in which bioengineering capstone design students provided formative feedback to their peers on figures and tables in reports.25 Like u.s., they taught a workshop on the principles of visual design and had students utilize them to critique the piece of work of their peers. This approach resulted in more informative effigy captions, although it is not clear whether the improvements extended to data estimation. They besides observed improvements in clarity and readability, just not statistically significant changes.
The pedagogical intervention described above has of import features that promote pupil learning. The "lecture" menses that addressed the LOs was designed based on active learning strategies, including cooperative learning, which take been shown to be successful in promoting student learning and mastery of material.xi,19 In a meta-analysis of 117 studies and over xx,000 participants, Abrami asserts that a "mixed" intervention of (1) guided instruction on critical thinking and (two) immersion of critical thinking in the discipline has more impact on student performance than either intervention given independently.2 During the intervention, nosotros taught using a "mixed" approach by combining authentic work in a tissue culture lab, direct teaching on visual communication, and practise on creating and critiquing materials.
In the intervention, nosotros gave explicit instruction on visual communication skills to students by guiding them to do information visualization and estimation in small peer groups. After the intervention, we saw significant improvements in peer- and self-evaluation and operation in LO1 (Evolution of Graphs and Tables) for the majority of the categories (specifically, eight of nine comparisons). We postulate that these meaning improvements are related to the nature of the intervention assignment. Students created their ain graphs or tables for this activeness and then had the opportunity to critique their work in pocket-sized groups. We speculate that this level of personal investment in the graph, the dialogue in which they engaged well-nigh their work, and the noesis they could brand improvements to it for their graded poster draft contributed to such substantial growth effectually this LO.
On the other hand, the intervention did not outcome in significant improvements in peer- and self-evaluation and performance for LO2 (Interpreting and Summarizing Results). In fact, improvements were seen in only three of 15 comparisons. Overall, students did not become noticeably better or worse at interpreting or summarizing experimental results. One possible reason for this may have been the nature of the intervention, where students wrote key results and titles based on tables and graphs developed by the teacher. Nosotros postulate that students' level of date in this activeness may accept macerated because they were non reviewing and revising their own work. Another possible reason is that the time devoted to this action (20–25 min) was insufficient for students to make substantial progress on this hard chore. Furthermore, students may take not take gained additional proficiency in interpreting experimental results because 1 50-min lecture covering technical content on tissue culture experiments was eliminated. On the other hand, the reduction in technical content appeared to have no detrimental effect. To ameliorate the outcomes related to LO2, we call back it would exist beneficial to spend more time on interpreting and summarizing results and to employ the students' own piece of work instead of instructor-generated examples.
As an important command, we considered impacts to Statements ix–13 that were not linked to either LO (Tabular array4). Since the majority (xiii of 15 comparisons) did not testify significant improvement, we can conclude that an intervention that targets one aspect of student learning does not positively or negatively touch on student performance in other areas of the assignment. That said, ii of 15 comparisons, namely Statements 12 and 13 for RQ1 (PvI), did return meaning results, but without convergence. This ways that as compared to the instructor, the peer evaluations got worse after the intervention for these ii statements. We cannot readily explain this. Still, we did not see whatsoever significant divergence for Statements 12 and thirteen for RQ2 or RQ3. If we had seen a similar reject for RQ3, which is related to performance, nosotros would have been concerned; however, this was not the instance.
Correlation between Student and Instructor Ratings Improved
Moving forward, we institute that student and kinesthesia ratings were more closely aligned after the intervention. The overall grades given by peers and instructors (Fig.twoa–b) presented several interesting trends. In Fig.two, the slope is less than i (where one would be perfect agreement) and the intercept is between four and 5, suggesting that overall grades from peers are higher than the instructor's. This result was consistent with our previous work that showed that students overestimate the quality of their peers' work.26 After the intervention, the correlation coefficient increased from r = 0.51 (Fig.2a, pre-intervention) to r = 0.68 (Fig.2b, mail-intervention), demonstrating a small improvement. The scatterplot and fits of the overall grades of SvI are not shown, simply we observed like trends, with r = 0.xviii for pre-intervention and r = 0.37 for post-intervention. As shown in our previous work, the correlation coefficients were lower for self-evaluation than for peer-evaluation.26
The mail service-intervention results are likewise consistent with Falchikov and Goldfinch's impressive meta-analysis of 48 peer-reviewed studies, from which they concluded that a strong correlation (mean r = 0.69) exists between instructors and peers in determining global ratings.x Based on their work10 and ours, we suggest factors that tin ameliorate student/instructor alignment and, ultimately, pupil functioning: increased guided instruction time on challenging tasks, a collaborative learning environment, and increased exercise by students where they utilize key principles in the domain of interest.
Acknowledgments
Nosotros would like to acknowledge Dr. Qiwei Li for his aid with the statistical analysis and Dr. Jordan Trachtenberg for her contributions to the manuscript.
Disharmonize of involvement
No benefits in whatsoever form have been or will exist received from a commercial party related directly or indirectly to the subject of this manuscript.
References
1. Abercrombie S, Parkes J, McCarty T. Motivational influences of using peer evaluation in problem-based learning in medical education. Interdiscip. J. Probl. Based Learn. 2015;9:33–43. [Google Scholar]
two. Abrami PC, Bernard RM, Borokhovski E, Wade A, Surkes MA, Tamim R, Zhang D. Instructional interventions affecting critical thinking skills and dispositions: a stage 1 meta-analysis. Rev. Educ. Res. 2008;78:1102–1134. doi: ten.3102/0034654308326084. [CrossRef] [Google Scholar]
iii. Agresti A. Adding constants to cells of a contingency table. In: Balding DJ, Bloomfield P, Cressie NAC, Fisher NI, Johnstone IM, Kadane JB, Ryan LM, Scott DW, Smith AFM, Teugels JL, editors. Categorical Data Assay. 2. Hoboken, NJ: Wiley; 2002. [Google Scholar]
4. Bedward, J., Due east. N. Wiebe, Fifty. Madden, M. Carter, and J. Minogue. Graphic literacy in simple science education: enhancing inquiry, applied science problem-solving and reasoning skills. ASEE Annual Conference & Exposition; 2009; Austin, TX, 2009. https://peer.asee.org/4719.
5. Berry FC, Carlson PA. Assessing technology blueprint experiences using calibrated peer review. Int. J. Eng. Educ. 2010;26:1–8. [Google Scholar]
six. Carlson, P. A., A. A. Russell, West. N. Waggenspack, C. G. Wilmot, D. F. Bowles, D. R. Voltmer, W. T. Monroe, Due west. N. Hull, and C. D. Raubenheimer. Improving engineering science education with enhanced calibrated peer review: assessment of a collaborative research project. ASEE Annual Conference & Exposition, San Antonio, TX, 2012. https://peer.asee.org/21501.
7. Carter M, Ferzli Yard, Wiebe Due east. Pedagogy genre to english kickoff-language adults: a study of the laboratory report. Inquiry in the Instruction of English. 2004;38:395–419. [Google Scholar]
8. Dupen, B. Educational activity graphical data presentation techniques in an introductory materials form. ASEE Annual Conference & Exposition, Austin, TX, 2009. https://peer.asee.org/5042.
9. Ernst, J., D. Lane, and A. Clark. Pictorial visual rotation ability of engineering blueprint graphics students. ASEE Annual Briefing & Exposition, Indianapolis, IN, 2014. https://peer.asee.org/22915.
10. Falchikov N, Goldfinch J. Pupil peer assessment in higher pedagogy: a meta-analysis comparing peer and teacher marks. Rev. Educ. Res. 2000;lxx:287–322. doi: 10.3102/00346543070003287. [CrossRef] [Google Scholar]
11. Freeman Southward, Boil SL, McDonough M, Smith MK, Okoroafor Due north, Jordt H, Wenderoth MP. Active learning increases pupil performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. USA. 2014;111:8410–8415. doi: 10.1073/pnas.1319030111. [PMC gratuitous article] [PubMed] [CrossRef] [Google Scholar]
12. Furman B, Robinson Due west. Improving engineering report writing with calibrated peer reviewtm. Front. Educ. 2003 [Google Scholar]
xiii. Gobin As, West JL. Val-ala-pro-gly, an elastin-derived not-integrin ligand: smooth muscle cell adhesion and specificity. J. Biomed. Mater. Res. A. 2003;67:255–259. doi: ten.1002/jbm.a.10110. [PubMed] [CrossRef] [Google Scholar]
14. Gunersel A, Simpson N. Improvement in writing and reviewing skills with calibrated peer reviewtm. Int. J. Scholarsh. Teach. Learn. 2009;iii:ane–14. [Google Scholar]
xv. Hartberg Y, Gunersel A, Simpson N, Balester V. Development of educatee writing in biochemistry using calibrated peer review. Journal of the Scholarship of Teaching and Learning. 2008;2:29–44. [Google Scholar]
16. Heise, Eastward. A., A. Palmer-Julson, and T. G. Su. Calibrated peer review writing assignments for introductory geology courses. Abstracts with Programs (Geological Society of America), Denver, CO, 2002. https://gsa.confex.com/gsa/2002AM/webprogram/Paper40755.html.
17. Mynlieff K, Manogaran A, St. Maurice M, Eddinger T. Writing assignments with a metacognitive component raise learning in a large introductory biology grade. CBE Life Sci. Educ. 2014;13:311–321. doi: 10.1187/cbe.thirteen-05-0097. [PMC free article] [PubMed] [CrossRef] [Google Scholar]
eighteen. National University of Engineering . The engineer of 2020: visions of engineering in the new century. Washington, DC: The National Academies Press; 2004. [Google Scholar]
19. Prince Grand. Does active learning work? A review of the research. J. Eng. Educ. 2004;93:223–231. doi: 10.1002/j.2168-9830.2004.tb00809.x. [CrossRef] [Google Scholar]
twenty. Sandi-Urena S, Cooper MM, Gatlin TA, Bhattacharyya G. Students' experience in a general chemical science cooperative problem based laboratory. Chem. Educ. Res. Pract. 2011;12:434–442. doi: 10.1039/C1RP90047A. [CrossRef] [Google Scholar]
21. Saterbak, A., Yard. Embree, and Yard. Oden. Client-based projects in freshman pattern. ASEE Annual Conference & Exposition, San Antonio, Texas, 2012. https://peer.asee.org/21074.
22. Saterbak, A., and T. Volz. Implementing calibrated peer review to raise technical critiquing skills in a bioengineering laboratory. ASEE Almanac Conference & Exposition; 2008; Pittsburgh, PA, 2008. https://peer.asee.org/3128.
23. Saterbak, A., and T. Volz. Assessing knowledge and application of the design process. ASEE Annual Conference & Exposition, Indianapolis, IN, 2014. https://peer.asee.org/20094.
24. Sherwood, J., and P. Avitabile. A existent world feel using linkages to teach design, analysis, cad and technical writing. ASEE Annual Conference & Exposition, Honolulu, HI, 2007. https://peer.asee.org/2954.
25. Taylor, A., and S. Pulford. Visual communication learning through peer design critiques: technology advice across divisions. ASEE Annual Conference & Exposition, Seattle, WA, 2015. https://peer.asee.org/25043.
26. Volz, T., and A. Saterbak. Students' strengths and weaknesses in evaluating technical arguments as revealed through implementing calibrated peer review™ in a bioengineering laboratory. [special event on writing techologies and writing across the curriculum]. Across the Disciplines. 6, 2009. https://wac.colostate.edu/atd/technologies/volz_saterbak.cfm.
27. Waggenspack, W., Due south. Liggett, Westward. Hull, D. Bowles, S. Sears, D. Thomas, and P. Davis. Incorporating visual communications assignments to enrich education in all engineering disciplines. ASEE Annual Briefing & Exposition, Louisville, KY, 2010. https://peer.asee.org/16090.
28. Walvoord K, Hoefnagels Grand, Gaffin D, Long D. An analysis of calibrated peer review (cpr) in a science lecture classroom. J. Coll. Sci. Teach. 2008;37:66–73. [Google Scholar]
livingstone-learmonthhappled.blogspot.com
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5809568/