TANAKA Sigeto <http://tsigeto.info/misconduct/>Human Reproduction, eLetter (submitted but not accepted) (2016-09-08)
(Tohoku University)
[PDF version] [On my persomal blog] [On Academia.edu] [OSF preprint]
[Communication] [Links]
I read an article about International Fertility Decision-making Study (IFDMS) by Bunting, Tsibulsky, and Boivin (2013) in Human Reproduction 28(2). Since it was based on data from a large-scale survey conducted in more than 18 countries with 13 languages (in 2009-2010), I was interested in understanding whether it yielded results that were valid and comparable across lingual and socio-cultural differences. After examining the article and other documents, I concluded that the IFDMS does not have a substantial level of validity and comparability. This letter explains why.
Bunting et al. (2013) reported some problems in the measurement. For the Cardiff Fertility Knowledge Scale (CFKS), one of the main topics of the survey, the standardized Cronbach alpha was low in Italy and Turkey (p. 387). This suggests failed measurements due to translation into Italian and Turkish.
In the 'Translations' section of the article (p. 388), there are problems in the process of designing multilingual questionnaires. The questions were framed in English, pre-tested by potential respondents, and then translated. In this process, questions might not have been selected to ensure comparability among different social conditions. Indeed, according to the Appendix, the second item of the CFKS includes a statement 'A couple would be classified as infertile if they did not achieve a pregnancy after 1 year of regular sexual intercourse (without using contraception)' and sets the right answer as 'TRUE'. However, the answer should have depended on the definition of infertility. In Japan in those days, for instance, the standard definition by the Japan Society of Obstetrics and Gynecology (2008, p. 276) applied the term 'infertility' to couples who had not conceived after regular intercourse of 2 years. The item thus failed to capture the existing varieties in the definition of the term.
The questionnaire-making process had another problem; the translated versions were not pre-tested by potential respondents. This could have resulted in unreadable or unanswerable questions being retained in the questionnaire.
This doubt should be confirmed by examining the questionnaires. However, there is no published IFDMS questionnaire. Although the article introduces two URLs, http://www.startingfamilies.com and http://www.startingfamilies.org, the former has expired while the latter is redirected to a webpage carrying no information on IFDMS. A published report on IFDMS (Merck Serono, No date) also contains no questionnaire.
I contacted the correspondence address of the article and directly got the Japanese version of the questionnaire (PDF files for male and female respondents). I have already reported about it in Japanese (Tanaka, 2016a, 2016b).
As an overall impression, the quality of translation is poor. There are many unnatural Japanese phrases among the 17 pages of the questionnaire, as well as spelling errors (e.g., 'atana' instead of 'anata' [=you]). Although different questionnaires were prepared for women and men, they are almost identical. As a result, some questions are unanswerable for male respondents: the questionnaire for men includes questions on the respondent's thought about why he is not pregnant.
There are mistranslations among the 13 items of the CFKS. Many English expressions involving comparisons and tendencies (e.g., 'more', 'never', and 'likely to') were not translated. For some items, Japanese postpositional particles were misused to make unintended connotations. In questions about male fertility, the terms 'fertility' and 'fertile' were translated into a Japanese word that meant artificial insemination. In total, I evaluated 10 of the 13 items to be improperly translated (Tanaka, 2016a).
In addition, the order of the 13 items of the CFKS differs between the English and Japanese versions. It is an elementary mistake not to keep the identical order of a series of questions. This would damage the comparability among questionnaires of different versions, due to the difference in carryover effects.
Regarding the other focus of the article, the six items on fertility treatment beliefs, the quality of translation is somewhat better than that of CFKS. Notwithstanding, there is also an obvious mistranslation: the auxiliary verb 'can' was not translated in the statement 'fertility treatments can cause emotional problems'.
I only read the Japanese questionnaire. I have no information on the other versions. However, as I mentioned above, it is probable that questions were mistranslated in Turkish and Italian. There is no evidence to indicate that the questionnaires in other languages are of better quality than the Japanese one. I accordingly expect no reliable research results with IFDMS, given that the data are unreliable.
As a professional social survey researcher, I am afraid of the spreading of unreliable information based on low-quality surveys. Researchers should make efforts to keep the quality of a survey and reject unreliable research results.
I am not sure whether the reviewers checked the IFDMS questionnaires. But, even if they did, it would be difficult to evaluate all questions in 13 languages. Generally speaking, it is unrealistic for journal editors to appoint native-level speakers for a dozen of languages to review a manuscript.
To check the quality of such multilingual study, the only realistic solution is readers' critical evaluation after the publication. For such evaluation, questionnaires and detailed documents should be accessible. In a traditional way, researchers compile questionnaires and details of the survey in a book-form report and donate it to libraries. With the progress of information technology today, it is convenient to publish them as a supplementary material on the journal website. I hope that the Human Reproduction editorial board will consider it.
TANAKA Sigeto
Associate Professor at Tohoku University, Graduate School of Arts and Letters
Sendai 980-8576 JAPAN
http://tsigeto.info/office.html
I am a member of activists' network against the use of unscientific knowledge in schools (Koukou Hoken Hukukyouzai no Siyou Tyuusi Kaisyuu o Motomeru Kai: http://fukukyozai.jimdo.com/)
Questions/comments are welcome.
[PDF version] [On my persomal blog] [On Academia.edu] [OSF preprint]
Tohoku Univ / School of Arts and Letters / Applied Japanese Linguistics / TANAKA Sigeto
History of this page:
Generated 2017-06-22 15:50 +0900 with Plain2.
Copyright (c) 2017 TANAKA Sigeto