| spelling |
oapen-20.500.12657-749212023-08-03T17:59:39Z Chapter An experimental annotation task to investigate annotators’ subjectivity in a Misogyny dataset Tontodimamma, Alice Ignazzi, Elisa Anzani, Stefano Stranisci, Marco Basile, Valerio FONTANELLA, Lara subjectivity misogyny disagreement social bias bic Book Industry Communication::J Society & social sciences In recent years, hatred directed against women has spread exponentially, especially in online social media. Although this alarming phenomenon has given rise to many studies both from the viewpoint of computational linguistics and from that of machine learning, less effort has been devoted to analysing whether models for the detection of misogyny are affected by bias. An emerging topic that challenges traditional approaches for the creation of corpora is the presence of social bias in natural language processing (NLP). Many NLP tasks are subjective, in the sense that a variety of valid beliefs exist about what the correct data labels should be; some tasks, for example misogyny detection, are highly subjective, as different people have very different views about what should or should not be labelled as misogynous. An increasing number of scholars have proposed strategies for assessing the subjectivity of annotators, in order to reduce bias both in computational resources and in NLP models. In this work, we present two corpora: a corpus of messages posted on Twitter after the liberation of Silvia Romano on the 9th of May, 2020 and corpus of comments constructed starting from posts on Facebook that contained misogyny, developed through an experimental annotation task, to explore annotators’ subjectivity. For a given comment, the annotation procedure consists in selecting one or more chunk from each text that is regarded as misogynistic and establishing whether a gender stereotype is present. Each comment is annotated by at least three annotators in order to better analyse their subjectivity. The annotation process was carried by trainees who are engaged in an internship program. We propose a qualitative-quantitative analysis of the resulting corpus, which may include non-harmonised annotations. 2023-08-03T15:06:52Z 2023-08-03T15:06:52Z 2023 chapter ONIX_20230803_9791221501063_117 2704-5846 9791221501063 https://library.oapen.org/handle/20.500.12657/74921 eng Proceedings e report application/pdf Attribution 4.0 International 9791221501063-49.pdf https://books.fupress.com/doi/capitoli/979-12-215-0106-3_49 Firenze University Press, Genova University Press ASA 2022 Data-Driven Decision Making 10.36253/979-12-215-0106-3.49 10.36253/979-12-215-0106-3.49 9223d3ac-6fd2-44c9-bb99-5b98ca9d2fad 863aa499-dbee-4191-9a14-3b5d5ef9e635 9791221501063 134 6 Florence open access
|
| description |
In recent years, hatred directed against women has spread exponentially, especially in online social media. Although this alarming phenomenon has given rise to many studies both from the viewpoint of computational linguistics and from that of machine learning, less effort has been devoted to analysing whether models for the detection of misogyny are affected by bias. An emerging topic that challenges traditional approaches for the creation of corpora is the presence of social bias in natural language processing (NLP). Many NLP tasks are subjective, in the sense that a variety of valid beliefs exist about what the correct data labels should be; some tasks, for example misogyny detection, are highly subjective, as different people have very different views about what should or should not be labelled as misogynous. An increasing number of scholars have proposed strategies for assessing the subjectivity of annotators, in order to reduce bias both in computational resources and in NLP models. In this work, we present two corpora: a corpus of messages posted on Twitter after the liberation of Silvia Romano on the 9th of May, 2020 and corpus of comments constructed starting from posts on Facebook that contained misogyny, developed through an experimental annotation task, to explore annotators’ subjectivity. For a given comment, the annotation procedure consists in selecting one or more chunk from each text that is regarded as misogynistic and establishing whether a gender stereotype is present. Each comment is annotated by at least three annotators in order to better analyse their subjectivity. The annotation process was carried by trainees who are engaged in an internship program. We propose a qualitative-quantitative analysis of the resulting corpus, which may include non-harmonised annotations.
|
