Abstract: Researchers use Inter-Rater Reliability (IRR) to measure whether two processes— people and/or machines—identify the same properties in data. There are many IRR measures, but regardless of the measure used, however, there is a common method for estimating IRR. To assess the validity of this common method, we conducted Monte Carlo simulation studies examining the most widely used measure of IRR: Cohen’s kappa. Our results show that the method commonly used by researchers to assess IRR produces unacceptable Type I error rates.

Abstract: Portfolio assessment gains new traction in youth-serving maker-educational spaces through increased inclusion of maker portfolios in college and job applications. However, the collaborative and cooperative character of making poses a tension to traditional portfolio assessment that is focused on showcasing individual achievements. Together, this calls for an expanded understanding of the use of portfolios in maker education. We examined the types of portfolio entries at two youth-serving makerspaces (one out-of-school and one in-school), and observed the documentation of personal and shared projects in personal and shared portfolios. Our main findings are that, compared to portfolios that focused on personal work alone, portfolios that included shared projects and documentation presented richer showcases, showing technical and social engagement, assessment by people across a distributed community, and possibilities to narrate work to multiple audiences. This has implications on the facilitation of maker portfolios and broadens portfolio assessment to show the role of the learner in society.

Abstract: During collaborative learning, computer-supported or otherwise, students balance taskoriented goals with the interpersonal goals of relationship-building; these goals may conflict, negatively impacting learning. In peer tutoring, for instance, tutors may avoid providing feedback to their partners to avoid the face-threat to their tutee. In this paper, we explore how the interpersonal closeness between tutor and tutee impacts tutors’ use of indirectness with feedback and instructions, and the impact those moves have on tutees’ problem-solving. We found that stranger tutors use more indirect instructions and provide more positive feedback to their tutee than friend tutors, and that stranger tutees attempted and solved more problems when their tutors used indirect instructions. We found no effect for dyads of friends, suggesting that interpersonal closeness reduces the face-threat of direct instructions. These results demonstrate that designers of CSCL tools should incorporate awareness of students’ relationships into their systems, as that relationship impacts students’ collaborative learning behaviors.

Abstract: While the maker movement and its associated affordable and accessible practices and toolkits have reinvigorated interest in pre-collegiate STEM, invention and creativity, many have critiqued makerspaces as implicitly exclusionary, particularly across gender, race and ethnicity. In an effort to rectify past participatory inequities, we designed a maker workshop for high school youth that capitalized upon multiple digital and physical interfaces to create simultaneously digitally and physically responsive projects, which encouraged teambased distributed creativity and development. We explore how the tools and the curricular design encouraged and fostered collaboration and inclusivity, as well as disrupted previous implicit associations around computing and creativity. We discuss the teams, the projects created and how the learning activities provided opportunities for inclusion and equity.

Abstract: Collaborative and individual learning are both frequently used in classrooms to support learning. However, little research has investigated the benefits of combining individual and collaborative learning, as compared to learning only individually or only collaboratively. With our study, we address this research gap. We compared a combined condition to individual-only or collaborative-only learning conditions using intelligent tutoring systems for fractions. The study was conducted with 382 4 and 5 grade students. Students across all three conditions had significant learning gains. However, the combined condition had higher learning gains than the individual or collaborative condition. This difference was more pronounced for 4 grade students than for 5 grade students. In addition, we found that students in the combined condition expressed higher situational interest in the activity compared to those working individually and the same as students working only collaboratively. Through a combination, we may support better student learning.

Abstract: Within peer collaboration, both cognitive and social phenomena have been identified as important components for success, though little is known about the relationship between these factors. In this work, we examined math collaboration discourse between 11 4t h grade dyads in 30-second slices to investigate the relationship between rapport state and reasoning state. Prior to collaboration, students watched one of three instructional videos modeling either domain knowledge, collaborate reasoning, or both. There was no impact of video type on student talk behaviors, nor posttest scores. However, we found a correlation between high rapport states and strong reasoning states, as well as a marginal effect of more co-constructive reasoning leading to improved posttest scores. This work demonstrates that students’ rapport states may play a role in students’ reasoning states, and thus calls for a deeper investigation within the CSCL community about the role of rapport in peer learning.

Abstract: To support learners during collaborative problem solving, developing a deeper understanding of collaborative dialogue is essential. This paper focuses on one important aspect of collaborative dialogue: expressions of uncertainty. In a study of undergraduate novice computer science students working in pairs, we observed that the students who produced the lowest quality solutions expressed uncertainty more often than those who produced middle-quality solutions. Perhaps surprisingly, pairs with the highest quality solutions also expressed more uncertainty than the middle performers. Examining the ways in which students expressed and then followed up on uncertainty revealed that higherperforming pairs utilized emerging learning opportunities when uncertainty was expressed, and remained focused on one task at a time. In contrast, the lower-performing pairs often did not resolve their uncertainty before moving on, attempting to work with multiple incomplete pieces of the solution simultaneously. These findings provide insight into how best to support collaborative learning during uncertainty.

Abstract: Working on students’ authentic problems is emphasized in Knowledge Building theory and pedagogy, as it is perceived that a failure to deal with such problems may result in a failure of knowledge building. This study is focused on questions students asked in a knowledge building environment, in order to examine how issues students cared enough about to pose as questions help knowledge building succeed. Comparing question threads (threads started with questions) and non-question threads (threads that did not start with questions), we noticed that problems posted by students engaged the community in a sustainable and progressive discourse, which is central to collaborative knowledge building. Moreover, the quality analysis of the data revealed that the threads starting with questions were more likely to end up with productive threads compared to the non-question threads.

Abstract: This paper describes our efforts to add structure and formalism to the design of a CSCL curriculum for high school science–integrating individual, collaborative and whole-class inquiry activities into a coherent “learning community.” A pedagogical model called Knowledge Community and Inquiry (KCI) guided our design of a curricular sequence in which one activity feeds into the next, responding differentially to students, and scaffolding new forms of interaction. We include real-time analysis of student interaction data as a source of input into the orchestration of complex scripts, which can influence the assignment of students to groups, the distribution of materials or sequencing of activities. It can also be used to determine which groups may need help, to provide groups with formative feedback, and to provide the instructor with information concerning student groups. The primary outcome of this paper is the design itself, which is evaluated in terms of its theoretical coherence.

Abstract: In a new era of digital media and democracy, there is widespread concern that technologies have incapacitated us from learning and teaching across diverse communities and perspectives. While this notion may ring true in certain contexts, this paper describes a study, “Mobile City Science,” that designed a novel learning experience in which educators and young people used mobile and place-based technologies to document and analyze the diverse perspectives of community members living in rapidly changing urban areas. The objective of this work was to teach young people digital literacies associated with “city science,” an emerging interdisciplinary field that creates data-driven approaches to complicated community issues. Participants were videotaped as they collected and analyzed information about a specific neighborhood using wearable cameras, GPS devices, heart rate monitors, and a GIS software. Early findings show that Mobile City Science uses technology to engage people with diverse perspectives around a community scale problem. Major issues addressed Technology has changed the nature of political engagement. For every “success” story of increased government transparency and youth mobilization, there is an instance of political balkanization and divisiveness (e.g., Manjoo, 2016). In this new era of digital media and technology, many have argued that the sheer ubiquity of our technological interactions have incapacitated us from learning and teaching across diverse communities and perspectives. In an article titled, “How We Broke Democracy,” Rose-Stockwell wrote, “If we cannot build the tools of our media to encourage empathy and consensus, we will retract further into toxic divisions that have come to define us today” (2016). This study, “Mobile City Science,” represents one such attempt to use technology and digital media as a means of encouraging empathy and building consensus around a “live” community problem. Importantly (and in contrast to Facebook and Twitter), technologies in Mobile City Science engage young people and youth educators in neighborhoods, in face-to-face interactions, to generate new information and representations of diverse perspectives. Before the most recent presidential election, social science was promoting the idea that “big data,” produced by our multiple devices and technologies, would make cities and their citizens “smarter,” or work better together. After the results of the election came through on November 8, 2016, several fundamental questions are now being asked about the promise of big data. Whose lives do these data actually represent? Who learns what from these data? What is the origin of data, and who has legitimacy to make interpretations and arguments from it? How did communities become so balkanized and bifurcated, bolstered by “data?” While these questions remain problematic issues for political engagement at large, they also open-up novel teaching and learning opportunities for underrepresented young people to create and engage with vast amounts of data across stakeholders that may have divergent perspectives on community issues. Creating insights and data-driven approaches to community issues, or “city science,” is an interdisciplinary field that is emerging alongside ubiquitous computing (MIT Media Lab, n.d.). Geospatial applications and mobile devices are especially conducive to this kind of data-driven inquiry process; these tools support and promote moving around the community and interfacing with shopkeepers, residents, and visitors in place. The mapping capabilities of geospatial apps and tools support a spatio-temporal way of recording, interpreting, arguing from the data collected around the neighborhood. These technological affordances have been shown to differently engage people in community-based issues, in ways that leverage physical mobility, be it walking, bussing, or bicycling around a geographic area (e.g., Nold, 2009; Taylor, 2013). Potential significance of the work As a whole, this research takes seriously the role of underrepresented young people using technology in Jane Jacobs’ (1961) provocation that “...cities have the capability of providing something for everybody, only because, and only when, they are created by everybody” (p. 238). Mobile City Science (MCS) provides four key contributions to the learning sciences and other fields concerned with new ways of engaging underrepresented youth in community-level issues and dialogue through technology. First, MCS informs and contributes to theories of embodied learning (e.g., Alibali & Nathan, 2012; Farnell, 1999; Glenberg, Gutierrez, Levin, CSCL 2017 Proceedings 391 © ISLS Japuntich, & Kaschak, 2004; Goldin-Meadow, Cook, & Mitchell, 2009) by analyzing how and what young people, and the people that educate them, learn about complex community issues from being on-the-move through their neighborhoods with mobile and location aware technologies. Second, MCS formalizes innovative ways of teaching community engagement to young people living in underserved areas of the city; MCS is a set of on-the-move teaching and learning experiences that support young people in collecting, analyzing, and arguing from spatial and other forms of data (e.g., GPS tracks, geo-referenced video files, density plots, placeelicited interviews). Again, these data represent the diversity of lived experiences within the geographic area. Third, MCS provides accessible, technologically enhanced ways for youth-serving organizations, community developers, urban planners, and/or social science educators to engage young people in civic processes and conversations happening at the scale of the city. Finally, MCS will contribute to a new theory of social change where technologies potentially democratize (rather than balkanize) learning and participation (e.g., Bilkstein, 2013; Papert, 1991; Resnick, et al., 2000; Wilensky & Papert, 2010) in processes of community development to include young people’s data-driven perspectives in planning and policy implementation.

Abstract: Advances in scripting theory and advances in support for student-driven knowledge construction call for a reconsideration of long-standing issues of guidance, control, and agency. This symposium undertakes a fresh analysis based on the relations between two widely adopted approaches that may be poles apart but arguably viewed as variations within a common applied epistemological framework. The two approaches are scripted collaboration and Knowledge Building. Rather than focusing on similarities and differences, the symposium will address deeper problems such as reconciling external supports of all kinds with the self-organizing character of knowledge construction and integrating such supports into classrooms viewed as knowledge-creating communities. The centerpiece of the symposium is a panel discussion that includes experts who provide different theoretical viewpoints. In its synthesis the symposium will capture and make sense of what is strongest in the two approaches and provide a broad conceptual basis for next-generation initiatives.

Abstract: Advances in scripting theory and advances in support for student-driven knowledge construction call for a reconsideration of long-standing issues of guidance, control, and agency. This symposium undertakes a fresh analysis based on the relations between two widely adopted approaches that may be poles apart but arguably viewed as variations within a common applied epistemological framework. The two approaches are scripted collaboration and Knowledge Building. Rather than focusing on similarities and differences, the symposium will address deeper problems such as reconciling external supports of all kinds with the self-organizing character of knowledge construction and integrating such supports into classrooms viewed as knowledge-creating communities. The centerpiece of the symposium is a panel discussion that includes experts who provide different theoretical viewpoints. In its synthesis the symposium will capture and make sense of what is strongest in the two approaches and provide a broad conceptual basis for next-generation initiatives.

Abstract: The contribution of this paper is twofold: we introduce a new kind of assessment developed to capture students’ learning in makerspaces, and we present a new perspective on how participating in a maker workshop impacts students. As opposed to traditional pen and paper tests, we designed a series of hands-on task that participants complete before and after a maker workshop. In this paper, we contrast high-school students’ performance with experts (graduate students in mechanical engineering) and found evidence that the students’ behavior became more similar to experts’ after participating in a maker workshop. For the scope of this paper, we focus on a single task and describe in detail our coding scheme and analyses. Additionally, we show how a combination of qualitative and computational analysis helped us develop metrics to compare novices’ and experts’ performances. We conclude by discussing the potential of this type of assessment for supporting students’ learning in makerspaces.

Abstract: This research aims to synthesize the extensive literature on Computer Supported Collaborative Learning (CSCL) in STEM education published between 2005-2014. Our synthesis focuses on the interactions of collaboration, technology, and pedagogies to see how different combinations may contribute to learning. A Latent Class Analysis was used to categorize existing research and points to a six-cluster solution. We have synthesized across and within the three largest clusters to 1) help us identify robust themes in this field and 2) help us better understand the positive outcomes associated with these aspects of CSCL in STEM education. The results suggest that different combinations of technology, pedagogy, and collaboration types require different strategies to scaffold students’ learning. This research provides a frame for synthesizing the effects of CSCL in synchronous and asynchronous STEM education and with various technologies and pedagogical designs. Visions of technology as a social entity are now ubiquitous and tout collaborative learning as a key benefit (Roschelle, 2013). There is extensive research regarding the use of CSCL in science, technology, engineering, and mathematics (STEM) education (Stahl et al., 2014; Jeong, Hmelo-Silver, & Yu, 2014). Many individual studies have reported encouraging results with different types of technologies used in a range of pedagogical approaches, with different forms of collaboration. Roschelle, Bakia, Toyama, and Patton (2011) have argued that we need to understand the “compound resources” at play in complex learning environments. However, there has been little systematic review on the impact of CSCL, especially ones focusing on the interactions of collaboration, technology, and pedagogies used (Jeong & Hmelo-Silver, 2016; Kirschner & Erkens, 2013). To understand the impact of CSCL research, it is important to examine the evidence based on the effectiveness of CSCL with both CSCL and measures of effectiveness broadly defined. Research in CSCL and the learning sciences should be especially well positioned to examine the complexity of these learning environments given the emphasis on mixed methods and design-based research (Jeong, Hmelo-Silver, & Yu, 2014; Roschelle et al., 2011), but as Kirschner and Erkens (2013) point out, we are not there yet. The editors of ijCSCL have recently noted that CSCL is becoming a mature field and that we need to understand the landscape of the field (Ludvigsen, Cress, Law, Rosé, & Stahl, 2016). Theoretical framework Our work uses meta-synthesis as a frame for reviewing research, allowing for the integration of research across qualitative and quantitative studies, and combinations of evidence across multiple studies (Suri & Clark, 2009; Bair, 1999). One of the strengths and challenges of this type of analysis is that it compares research that may not have common metrics. Research in CSCL focuses on learning as a cognitive and/or social process and studies learning designs, learning processes, and pedagogic practices that support technology-mediated collaborative processes in communities of practice. CSCL research is guided by a variety of theoretical frameworks that include information processing, social constructivist, sociocultural, social psychology, and communication theories (e.g., Jeong, Hmelo-Silver, & Yu, 2014; O’Donnell & O’Kelly, 1994). This synthesis cuts across theoretical frameworks to focus on how different combinations of technology, approaches to collaboration, and pedagogy contribute to learning, as these are the pillars of CSCL. Technologies that promote learning through collaboration mirror major shifts in education, which characterizes learning as being social and collective rather than individual. The changing pedagogies that support these perspectives and evolving technologies have merged to create many new CSCL opportunities in classrooms (Jeong & Hmelo-Silver, 2016; Miyake, 2007). Jeong and Hmelo-Silver (2016) have recently theorized that technology needs to have particular affordances to support CSCL. These include 1) establishing a joint task, 2) communication, 3) sharing resources, 4) engaging in productive processes, 5) engaging in co-construction, 6) monitoring and regulation, and 7) finding and building groups and communities. These affordances can be realized when the interactions of CSCL 2017 Proceedings 439 © ISLS technology, pedagogy, and modes of collaboration are considered. Our work considers this complex nature of CSCL and seeks synthesis within these interactions.

Abstract: Identifying the effects of students’ collaborative regulation behavior when working on a task is an important step towards a better understanding of how collaboration supports learning. We discuss a study where we combined analysis of students’ dialogues with an automated analysis of their action patterns as they constructed science models in an open-ended learning environment. Our results show that students use different types of collaborative regulation be-haviors, and that these behaviors affect their performance on the system as well as their pre-post learning gains. We also showed that groups, which adopt more shared regulation used different learning strategies than groups that did not.

Abstract: Effective collaborative behavior between students is neither spontaneous nor continuous. A system that can measure collaboration in real-time may be useful. For instance, it could alert an instructor that a group needs attention. We tested whether superficial measures of speech and user interactions of students would suffice for measuring collaboration. As pairs of students solved complex math problems on tablets, their speech and tablet gestures were recorded. These data and multi-camera videos were used by humans to code episodes as collaborative vs. various kinds of non-collaboration. Using just the speech and tablet log data, several detectors were machine learned. The best had an overall accuracy of 96% (Kappa=0.92), which is higher than earlier attempts to use speech and log data for detecting collaboration. The improved accuracy appears to be due both to our analytic methods and to the particular mathematical task, which involves moving objects.

Abstract: The idea of using gaze as a medium to look into the collaborative processes had been around in CSCL for past few years. However, it had not been widely used in the community. Most of the works done in the direction of understanding collaborative cognition are majorly based on the qualitative methods. Research has shown that the collaborative gaze data can be used as an alternate source of information to assess collaboration. Once, we understand the how the gaze data reflects the collaboration quality and success, we could design gaze-aware systems to support remote/collocated collaboration. In this symposium, we bring together five papers that use eye-tracking data as a proxy for communication and cognition during remote/collocated collaborative learning and propose design of gaze-aware systems.

Abstract: Working collaboratively holds many benefits for learners. However, varying incoming knowledge and attitudes toward collaboration present challenges and can lead to frustration for students. An important open question is how to support effective collaboration and foster equity for students with different levels of incoming preparation. In this study, we compared two collaborative instructional approaches for computer science problem solving, in which students participated in one of two conditions: The Baseline condition featured collaborative problem solving in which students worked in dyads from the beginning of the collaboration; in the other condition, called Think-First, students first worked on the problem individually for a short time and then began collaborating to produce a common solution. The results from 190 students from an introductory programming class working in 95 pair-programming teams demonstrate that this simple modification to pair programming had a significant positive effect on test scores and on substantive contributions in collaborative dialogue.

Abstract: Perspective-taking of a wide variety of pupils or students is fundamental in designing a dialogic classroom. As a vehicle of perspective-taking, tangible puppetry CSCL can create a learning environment that reduces the participants' anxiety or apprehension toward evaluation and draw out various types of pupils or students, allowing them to learn various perspectives. A classroom study revealed that the effect of tangible puppetry role-play remained in the immediate transfer task; the participants could elicit a variety of voices from possible pupils even in the self-performed role-play, and as well as on their essay. However, the mutual feedback discussions in the third session changed significantlyas similar to the first trial. This paper discusses necessary future directions to promote better reflection and to deepen perspective-taking through the tangible puppetry. Introduction Designing an effective lesson leveraging dialogic pedagogy is an essential skill for schoolteachers (Mutton, Hagger, & Burn, 2011)—but even for experienced teachers, it is difficult to operationalize in a classroom. In the dialogic classroom, teachers and students address learning tasks, listen to each other, share ideas and consider alternative viewpoints together. Students articulate their ideas freely—without fear of embarrassment over wrong answers— and help each other reach a shared understanding (Alexander, 2008). The teachers need to design a dialogue to stimulate the students’ thinking and advance their learning and understanding through structured and cumulative questioning and discussion, without monologic knowledge transmission. To prepare in designing a dialogue which ensures various students’ participation, the teachers need to imagine a wide variety of voices of their students and possible reactions and questions (Bahktin, 1981). Microteaching is one of the ways to practice how to implement dialogic pedagogy in teaching; however, it is not easy to achieve. One of the reasons discussed in the “apprenticeship of observation” framework (Lortie, 1975) is that student teachers and novices experienced monologic teaching as students themselves. However, we argue that there is another difficulty – excessive self-consciousness (Ladrousse, 1989) or evaluation apprehension (Cottrell et al., 1968) during microteaching. The role-play requires (student) teachers to act out young pupils in a realistic way which they may feel difficulty in, creating a tendency to play honest students who follow the teacher’s instruction without questioning. The past study discussed that tangible puppetry can serve as a powerful device for allowing people to overcome emotional or interpersonal obstacles in face-to-face role-play, and for eliciting reactions including inner emotions or unconscious experiences that they have had in a problematic situation (Mochizuki, et al., 2015). Puppetry allows each participant to obtain participant-observer balance by creating a clear separation between self (puppeteer) and non-self (puppet) as well as character (puppet) and observer (puppeteer) while playing a puppetry story, so that participants can use informal/irregular discourse more in the puppetry than in the case of normal self-performed role-plays where they rarely used informal/irregular one (Aronoff, 2005). We argued that puppetry can be a catalyst material to elicit and learn more realistic students’ reactions to foster perspective-taking of a wide variety of students, and developed a tangible puppetry CSCL system to help microteaching role-play in a puppetry format (Mochizuki et al., 2015). The system records the actions and conversations of the participants (hereinafter, the “character”) on top of a transparent table (Figure 1 (a)). In Figure 1, photo (a) shows the system ready to be implemented. Each puppet or prop is attached to a transparent box with an AR marker on the bottom. Each character can express his or her puppet’s condition by manipulating a switch to change the color of the LED in the box to either red or blue (Figure 1 (b)). A red LED may represent a sleeping/careless student, and blue an attentive/note-taking student. A web camera and microphone under the table record the puppets’ movements and conversations (i.e., the behavior of the characters), by detecting the CSCL 2017 Proceedings 593 © ISLS AR markers. After the role-play (Figure 1 (c)), the participants can view the recorded puppetry to inspire reflection (Figure 1 (d)). The webpage displays the role-play in animated form from a bird’s-eye view. The present study aims to examine the effectiveness of puppetry microteaching role-play, especially on perspective-taking. We demonstrated the preliminary evaluation of the CSCL system by comparison with selfperformed role play. This study examines an immediate transfer of perspective-taking training using the system so that we can discuss further promising ways to nurture the dialogic teaching skills. (a) (b) (c) (d) Figure 1. The CSCL system for tangible puppetry.

Abstract: Research has shown that the construction of visual representations may have a positive effect on cognitive skills, including argumentation. In this paper we present a study on learning argumentation through computer-supported argument diagramming. We specifically focus on whether students, when provided with an argument-diagramming tool, create better diagrams, are more motivated, and learn more when working with other students or on their own. We use learning analytics to evaluate a variety of student activities: pre and post questionnaires to explore motivational changes; the argument diagrams created by students to evaluate richness, complexity and completion; and pre and post knowledge tests to evaluate learning gains.

Abstract: Researchers use Inter-Rater Reliability (IRR) to measure whether two processes— people and/or machines—identify the same properties in data. There are many IRR measures, but regardless of the measure used, however, there is a common method for estimating IRR. To assess the validity of this common method, we conducted Monte Carlo simulation studies examining the most widely used measure of IRR: Cohen’s kappa. Our results show that the method commonly used by researchers to assess IRR produces unacceptable Type I error rates.

Abstract: Jianwei Zhang, University at Albany, SUNY, jzhang1@albany.edu Maria Bogouslavsky, University of Toronto, maria.bogouslavsky@gmail.com Guangji Yuan University at Albany, SUNY, gyuan@albany.edu , Abstract: This study explores cross-community interaction in two Grade 5/6 knowledge building communities. The two classrooms studied human body systems with the support of Knowledge Forum over a 10-week period. As the students conducted focused inquiry and discourse within their own community, they reviewed productive threads of ideas and posted syntheses in a cross-community space, as synthetic boundary objects. A set of idea thread syntheses from previous classrooms studying human body systems was also posted in the cross-community space. Qualitative analyses of classroom videos, online discourse, and interviews provide a rich description of how the students conceived, generated, and interacted around the synthetic boundary objects for knowledge building across communities.

Abstract: The way in which digital technologies take part and contribute to configuring teaching and collaborative learning practices has become a timely research matter in our field. Current studies in the ...