The Correlation of Machine Translation Evaluation Metrics with Human Judgement on Persian Language

PDF
  • Marziyeh Taleghani1,
  • ehsan pazouki*2,
  • Vahid Ghahraman3,
  1. MA in Translation Studies, Faculty of Persian Literature and Foreign Languages, South Tehran Branch of Azad University Iran
  2. Assistant Professor of Artificial Intelligence, Faculty of Computer Engineering, Shahid Rajaei Teacher Training University, Tehran, Iran
  3. Assistant Professor of TESOL, Iran Encyclopedia Compiling Foundation, Tehran, Iran

Revised: 2019-11-10

Accepted: 2019-11-10

Published in Issue 2019-07-01

How to Cite

Taleghani, M., pazouki, ehsan, & Ghahraman, V. (2019). The Correlation of Machine Translation Evaluation Metrics with Human Judgement on Persian Language. Journal of Language and Translation, 9(3), 43-55. https://oiccpress.com/ttlt/article/view/15693

PDF views: 190

Abstract

Machine Translation Evaluation Metrics (MTEMs) are the central core of Machine Translation (MT) engines as they are developed based on frequent evaluation. Although MTEMs are widespread today, their validity and quality for many languages is still under question. The aim of this research study was to examine the validity and assess the quality of MTEMs from Lexical Similarity set on machine translated Persian texts. This study focused on answering three main questions, which included the extent that Automatic Machine Translation Evaluation Metrics is valid on evaluating translated Persian texts; the probable significant correlation between human evaluation and automatic evaluation metrics in evaluating English to Persian translations; and the best predictor of human judgment. For this purpose, a dataset containing 200 English sentences and their four reference human translations, was translated using four different statistical Machine translation systems. The results of these systems were evaluated by seven automatic MTEMs and three human evaluators. Then the correlations of metrics and human evaluators were calculated using both Spearman and Kendall correlation coefficients. The result of the study confirmed the relatively high correlation of MTEMs with human evaluation on Persian language where GTM proved to be more efficient compared with other metrics.
PDF