Abstract
Sound design has been a fundamental component of audiovisual storytelling in linear media. However, with recent technological developments and the shift towards non-linear and immersive media, things are rapidly changing. More sensory information is available and, at the same time, the user is gaining agency upon the narrative, being offered the possibility of navigating or making other decisions. These new characteristics of immersive environments bring new challenges to storytelling in interactive narratives and require new strategies and techniques for audiovisual narrative progression. Can technology offer an immersive environment where the user has the sensation of agency, of choice, where her actions are not mediated by evident controls but subliminally induced in a way that it is ensured that a narrative is being followed? Can sound be a subliminal element that induces attentional focus on the most relevant elements for the narrative, inducing storytelling and biasing search in an immersive non-linear audiovisual environment? Herein, we present a literature review that has been guided by this prospect. With these questions in view, we present our exploration process in finding possible answers and potential solution paths. We point out that consistency, in terms of coherency across sensory modalities and emotional matching may be a critical aspect. Finally, we consider that this review may open up new paths for experimental studies that could, in the future, provide new strategies in the practice of sound design in the context of non-linear media.

Abstract
Since the beginning of the XXI century, we have been witnessing a significant shift in the media landscape towards enhanced immersive audiovisual manifestations, from controlled research environments to gradual production market penetration. Virtual reality, augmented reality, mixed reality, extended reality, 360 degrees video, and digital games are representative examples of these immersive technologies. Spatial audio design and production are instrumental to the immersive experience. As Ambisonics techniques do potentially mean more expense - in memory, processing power, and production budget -, limited exploration in the development of new composition and production methodologies across popular music production has been considered beyond the traditional stereophonic format. Our work details a post-production case study using spatial audio, namely High Order Ambisonics. The case study is a Brazilian popular song, remixed using 3rd order Ambisonics from a multitrack recording session composed of monophonic and stereophonic audio tracks. The song encompasses a unique approach for audio spatialization guided by hierarchical audio content attributes across multiple structural time scales and musical contexts. The evaluation of our production process adopted iterative heuristic assessments comparing technical decisions and aesthetic intentions in fostering an augmented spatial audio song. A set of technical guidelines and good practices on how and why to positioning audio in space are abstracted from our case study evaluation, which critically advances the theory and practice of popular musical audio production in immersive technologies.

Abstract
Professional theatre actors are highly specialized in controlling their own expressive behaviour and non-verbal emotional expressiveness, so they are of particular interest in fields of study such as affective computing. We present Acting Emotions, an experimental protocol to investigate the physiological correlates of emotional valence and arousal within professional theatre actors. Ultimately, our protocol examines the physiological agreement of valence and arousal amongst several actors. Our main contribution lies in the open selection of the emotional set by the participants, based on a set of four categorical emotions, which are self-assessed at the end of each experiment. The experiment protocol was validated by analyzing the inter-rater agreement (> 0.261 arousal, > 0.560 valence), the continuous annotation trajectories, and comparing the box plots for different emotion categories. Results show that the participants successfully induced the expected emotion set to a significant statistical level of distinct valence and arousal distributions. © 2022 Owner/Author.

Abstract
Digital musical instruments have become increasingly prevalent in musical creation and production. Optimizing their usability and, particularly, their expressiveness, has become essential to their study and practice. The absence of multimodal feedback, present in traditional acoustic instruments, has been identified as an obstacle to complete performer-instrument interaction in particular due to the lack of embodied control. Mobile-based digital musical instruments present a particular case by natively providing the possibility of enriching basic auditory feedback with additional multimodal feedback. In the experiment presented in this article, we focused on using visual and haptic feedback to support and enrich auditory content to evaluate the impact on basic musical tasks (i.e., note pitch tuning accuracy and time). The experiment implemented a protocol based on presenting several musical note examples to participants and asking them to reproduce them, with their performance being compared between different multimodal feedback combinations. Collected results show that additional visual feedback was found to reduce user hesitation in pitch tuning, allowing users to reach the proximity of desired notes in less time. Nonetheless, neither visual nor haptic feedback was found to significantly impact pitch tuning time and accuracy compared to auditory-only feedback.

Abstract
Emotional Machines is an interactive installation that builds affective virtual environments through spoken language. In response to the existing limitations of emotion recognition models incorporating computer vision and electrophysiological activity, whose sources are hindered by a head-mounted display, we propose the adoption of speech emotion recognition (from the audio signal) and semantic sentiment analysis. In detail, we use two machine learning models to predict three main emotional categories from high-level semantic and low-level speech features. Output emotions are mapped to audiovisual representation by an end-To-end process. We use a generative model of chord progressions to transfer speech emotion into music and a synthesized image from the text (transcribed from the user's speech). The generated image is used as the style source in the style-Transfer process onto an equirectangular projection image target selected for each emotional category. The installation is an immersive virtual space encapsulating emotions in spheres disposed into a 3D environment. Thus, users can create new affective representations or interact with other previous encoded instances using joysticks. © 2022 Owner/Author.

Abstract
AbstractWe advance Mixmash-AIS, a multimodal optimization music mashup creation model for loop recombination at scale. Our motivation is to (1) tackle current scalability limitations in state-of-the-art (brute force) computational mashup models while enforcing the (2) compatibility of audio loops and (3) a pool of diverse mashups that can accommodate user preferences. To this end, we adopt the artificial immune system (AIS) opt-aiNet algorithm to efficiently compute a population of compatible and diverse music mashups from loop recombinations. Optimal mashups result from local minima in a feature space representing harmonic, rhythmic, and spectral musical audio compatibility. We objectively assess the compatibility, diversity, and computational performance of Mixmash-AIS generated mashups compared to a standard genetic algorithm (GA) and a brute force (BF) approach. Furthermore, we conducted a perceptual test to validate the objective evaluation function within Mixmash-AIS in capturing user enjoyment of the computer-generated loop mashups. Our results show that while the GA stands as the most efficient algorithm, the AIS opt-aiNet outperforms both the GA and BF approaches in terms of compatibility and diversity. Our listening test has shown that Mixmash-AIS objective evaluation function significantly captures the perceptual compatibility of loop mashups (p < .001).