The multimodal approach in audiovisual translation

This paper will explore the multimodal approach to audiovisual translation (AVT). It must first be stressed, however, that most research on multimodality has not as yet focused on questions of translation. The Routledge Handbook of Multimodal Analysis (Jewitt 2009), which contains articles by most of the leading figures in the field, while representing a major step forward in multimodal studies, does not tackle translation head on. The word ‘translation’ does not even appear in the index. Over a relatively short time span, most of the major contributions to the field have been more purely linguistically based and intent on providing keys to the understanding of the interplay of semiotic resources such as words, images, gesture, music, light, etc. (see O’Toole 1994; Kress and van Leeuwen 1996; Martinec 2000; Unsworth 2001; Baldry and Thibault 2006, etc.). The work of these scholars, however, has provided an impetus to developing ideas on how to exploit multimodal analyses in the area of AVT. Thibault’s work, for example, on the ‘multimodal transcription’ provided this author with the basis for investigating how the integration of semiotic modalities in a film text could assist the subtitler in making those all-important decisions on what to retain and what to discard when faced with time constraints. Other scholars have studied the co-articulation of words and image in their discussion of how different modalities realize social functions and make meaning (O’Halloran 2008; Bednarek 2010), emphasising the importance of supplementing purely linguistic analyses with studies of all the other semiotic resources that make up a multimodal text. Findings will inevitably be reported verbally but the analyses need to explore the concept of integration and how other resources can interact with language and, crucially, how translators can be made sensitive to the entire semiotic impact of a multimodal text.