Abstract
Computer-supported online 3-D virtual world environments have been waxed and waned in interest and representativeness for supporting collaborative- and simulation-based practices. In a post-modern societal framework that requires inexpensive solutions for high-risk situations, research efforts in virtual worlds have developed a basis for understanding the use of virtual reality for multidisciplinary scenarios such as distance learning, training, therapy treatments, and social interaction. Complex relationships can be established simultaneously between several students functioning as integrated learning units using different media, and interacting with their physical environment in the context of real-world settings. In this sense, a recurrently updated research agenda for virtual worlds can characterize the current needs at a systematic way. This paper presents a meta-analysis of 35 publications to identify gaps and opportunities for research in collaborative three-dimensional environments based on content analysis. At a general perspective, there is a lack of established approaches to measure the influence and research potential of sociocultural factors in virtual worlds’ usage, autism spectrum and other healthcare-related settings, learning outcomes, content characteristics, task support for groups and crowds, and online data collection.

Abstract
Crowdsourcing has shown to be a valuable problem-solving approach to handle the increasing complexity and scale of tasks for which the current AI algorithms are still struggling. Crowd intelligence can be particularly useful to train and supervise AI systems in a symbiotic, co-evolutionary relationship that raises long-term research challenges to the hybrid, crowd-computing design space. With the increase in the scale of mixed-initiative approaches, we need to gain a better understanding of the implications of crowd-powered systems as a scaffold for AI through the study of massive crowd-machine interactions. In this paper, we identify some open challenges and design implications for future crowd-AI hybrid systems. A framework is also proposed based on the practical challenges of addressing human-centered AI methods and processes.

Abstract
A substantial amount of work is often overlooked due to the exponential rate of growth in global scientific output across all disciplines. Current approaches for addressing this issue are usually limited in scope and often restrict the possibility of obtaining multidisciplinary views in practice. To tackle this problem, researchers can now leverage an ecosystem of citizens, volunteers and crowd workers to perform complex tasks that are either difficult for humans and machines to solve alone. Motivated by the idea that human crowds and computer algorithms have complementary strengths, we present an approach where the machine will learn from crowd behavior in an iterative way. This approach is embodied in the architecture of SciCrowd, a crowd-powered human-machine hybrid system designed to improve the analysis and processing of large amounts of publication records. To validate the proposal's feasibility, a prototype was developed and an initial evaluation was conducted to measure its robustness and reliability. We conclude this paper with a set of implications for design.

Abstract
Researchers in a variety of fields are increasingly adopting crowdsourcing as a reliable instrument for performing tasks that are either complex for humans and computer algorithms. As a result, new forms of collective intelligence have emerged from the study of massive crowd-machine interactions in scientific work settings as a field for which there is no known theory or model able to explain how it really works. Such type of crowd work uses an open participation model that keeps the scientific activity (including datasets, methods, guidelines, and analysis results) widely available and mostly independent from institutions, which distinguishes crowd science from other crowd-assisted types of participation. In this paper, we build on the practical challenges of crowd-AI supported research and propose a conceptual framework for addressing the socio-technical aspects of crowd science from a CSCW viewpoint. Our study reinforces a manifested lack of systematic and empirical research of the symbiotic relation of AI with human computation and crowd computing in scientific endeavors.

Abstract
Over the past decade, online crowdsourcing has established itself as an emerging paradigm that industry and government have been using to harness the cognitive abilities of a multitude of users distributed around the world. In this context, microtask crowdsourcing has become the method of choice for addressing a wide range of diverse problems. Microtasks typically require a minimum of time and cognitive effort, but combined individual efforts have made it possible to accomplish great achievements. The goal of this paper is to contribute to the ongoing effort of understanding whether the same success that microtask crowdsourcing has achieved in other domains can be obtained in the field of social news curation. In particular, we ask whether it is possible to turn online news curation, typically a social and collaborative activity on the Web, into a model in which curatorial activities are mapped into microtasks to be performed by a crowd of online users.

Abstract
With cutting edge scientific breakthroughs, human-centred algorithmic approaches have proliferated in recent years and information technology (IT) has begun to redesign socio-technical systems in the context of human-AI collaboration. As a result, distinct forms of interaction have emerged in tandem with the proliferation of infrastructures aiding interdisciplinary work practices and research teams. Concomitantly, large volumes of heterogeneous datasets are produced and consumed at a rapid pace across many scientific domains. This results in difficulties in the reliable analysis of scientific production since current tools and algorithms are not necessarily able to provide acceptable levels of accuracy when analyzing the content and impact of publication records from large continuous scientific data streams. On the other hand, humans cannot consider all the information available and may be adversely influenced by extraneous factors. Using this rationale, we propose an initial design of a human-AI enabled pipeline for performing scientometric analyses that exploits the intersection between human behavior and machine intelligence. The contribution is a model for incorporating central principles of human-machine symbiosis (HMS) into scientometric workflows, demonstrating how hybrid intelligence systems can drive and encapsulate the future of research evaluation.