Published on 12.02.20 in Vol 3, No 1 (2020): Jan-Dec
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/15643, first published Jul 25, 2019.
Comparison of Traditional Citation Metrics and Altmetrics Among Dermatology Journals: Content and Correlational Analysis Study
Background: Research impact has traditionally been measured using citation count and impact factor (IF). Academics have long relied heavily on this form of metric system to measure a publication’s impact. A higher number of citations is viewed as an indicator of the importance of the research and a marker for the impact of the publishing journal. Recently, social media and online news sources have become important avenues for dissemination of research, resulting in the emergence of an alternative metric system known as altmetrics.
Objective: We assessed the correlation between altmetric attention score (AAS) and traditional scientific impact markers, namely journal IF and article citation count, for all the dermatology journal and published articles of 2017.
Methods: We identified dermatology journals and their associated IFs available in 2017 using InCites Journal Citation Reports. We entered all 64 official dermatology journals into Altmetric Explorer, a Web-based platform that enables users to browse and report on all attention data for every piece of scholarly content for which Altmetric Explorer has found attention.
Results: For the 64 dermatology journals, there was a moderate positive correlation between journal IF and journal AAS (rs=.513, P<.001). In 2017, 6323 articles were published in the 64 dermatology journals. Our data show that there was a weak positive correlation between the traditional article citation count and AAS (rs=.257, P<.001).
Conclusions: Our data show a weak correlation between article citation count and AAS. Temporal factors may explain this weak association. Newer articles may receive increased online attention after publication, while it may take longer for scientific citation counts to accumulate. Stories that are at times deemed newsworthy and then disseminated across the media and social media platforms border on sensationalism and may not be truly academic in nature. The opposite can also be true.
JMIR Dermatol 2020;3(1):e15643
Research impact has traditionally been measured using citation count and impact factor (IF). Academics have long relied on this form of metric system to measure a publication’s impact . A higher number of citations is viewed as an indicator of the importance of the research and a marker for the impact of the publishing journal [ , ]. More recently, social media and online news sources have become important avenues for dissemination of research, resulting in the emergence of an alternative metric system known as altmetrics [ , ]. Altmetrics is data that can explain both the nature and volume of attention that research receives. It measures how many people have engaged with and shared research and allows the researchers or publishers to see in what manner it was shared.
Numerous altmetrics harvesting tools have recently been developed, including the Altmetric Explorer. Altmetric Explorer is a Web-based platform that enables users to browse and report on all attention data for every piece of scholarly content for which the platform has found attention . The system tracks the online attention research receives by aggregating data from numerous avenues of online sharing sources, such as public policy documents, Mendeley, PubMed, mainstream media and social media outputs like Facebook and Twitter, and many more [ ]. An altmetric attention score (AAS), derived by an automated algorithm, indicates the amount of attention and, in some cases, public engagement that research has received [ ]. Therefore, articles that generate more attention are likely to have higher scores [ ]. The AAS is based on three main principles: Volume, sources, and authors [ ]. In addition, each form of mention contributes a different amount to the final score; for example, if an article is mentioned and shared via an online newspaper, it receives a higher score than if it is mentioned in a single tweet [ ].
The aim of this study was to investigate the relationship between altmetrics and traditional citation metrics among dermatology journals and articles. Does sharing research online via social media and other news sources correlate with article citation counts and journal IFs? We assessed the correlation between AAS and traditional scientific impact markers, namely journal IF and article citation count, for all the dermatology journals and published articles of 2017.
We identified dermatology journals available in 2017 and their associated IFs using InCites Journal Citation Reports . We entered all 64 dermatology journals into the Altmetric Explorer search platform, and the search year was limited to 2017. This generated the AAS for each journal and its published articles. Using Dimensions, an online research database that provides the current citation count for published research, we gathered the citation counts for each published article [ ]. We applied the Spearman rank correlation coefficient to assess the correlation between dermatology journal IF and AAS as well as dermatology article citation count and AAS. Statistical analysis was performed using SPSS V21 (IBM Corporation, Somers).
From the Journal Citation Report 2017, there were 64 dermatology journals with IFs ranging from 8.1 to 0.08. Of these journals, there was a moderate positive correlation between journal IF and journal AAS (rs=.513, P<.001;). From these journals, 6323 articles were published in 2017. Our data show a weak positive correlation between traditional citation count and AAS (rs=.257, P<.001; ).
Altmetrics complements, rather than replaces, the traditional citation metric system. Researchers and publishers can see not only how their research is distributed online but also in what form it is distributed, either through social media platforms such as Twitter and Facebook or via news articles. Our data demonstrate a moderate correlation between journal IF and AAS among dermatology journals, suggesting that the higher IF dermatology journals have a larger impact across social media and mainstream media. Our data also show a weak correlation between article citation count and AAS. Temporal factors may explain this weak association: Newer articles may receive increased online attention after publication, while it may take longer for scientific citation counts to accumulate. The weak correlation between the number of article citations and AAS in our study is consistent with recently published work of weak correlations between AAS and both cardiology and pediatric surgery citation counts [, ]. These findings suggest that articles achieving high AAS, and therefore online attention, are not of equivalent interest to academics.
It is also important to note the occasional disconnect between what are deemed newsworthy publications and comprehensive academic research. Altmetrics reflects online crowd attention but does not reflect the quality, validity, and originality of research. The obverse could be said about traditional citation metrics, which focus on quality and validity but not dissemination of new research. For these metrics, the journal reach is limited by subscription fees or firewalls.
The near instantaneous nature of the altmetric score, with the ability for an article to go viral and a corresponding rapid boost in the AAS, enables the gaming of altmetrics. With enough time, coupled with the speed of the internet and freedom to post, research AAS could be artificially inflated by repeated tweets and widespread dissemination of research online. However, altmetrics addresses this issue by weighing author contribution with each mention to the overall attention score.
A limitation of this study was the exclusion of non-dermatological journals such as Nature, New England Journal of Medicine, and the Lancet, which also publish highly cited dermatological articles.
To our knowledge, this is the first study to assess the relationship between dermatology journal citation metrics and AAS.
Conflicts of Interest
- Sud P, Thelwall M. Evaluating altmetrics. Scientometrics 2014;98(2):1131-1143. [CrossRef]
- Selvaraj S, Borkar DS, Prasad V. Media Coverage of Medical Journals: Do the Best Articles Make the News? PLoS ONE 2014 Jan 17;9(1):e85355. [CrossRef]
- O'Connor EM, Nason GJ, O'Kelly F, Manecksha RP, Loeb S. Newsworthiness vs scientific impact: are the most highly cited urology papers the most widely disseminated in the media? BJU Int 2017 May 18;120(3):441-454. [CrossRef]
- Patel RB, Vaduganathan M, Bhatt DL, Bonow RO. Characterizing High-Performing Articles by Altmetric Score in Major Cardiovascular Journals. JAMA Cardiol 2018 Dec 01;3(12):1249-1251. [CrossRef]
- Altmetric. 2018. URL: https://www.altmetric.com/about-our-data/how-it-works/ [accessed 2019-10-01]
- Heidenreich A, Bastian PJ, Bellmunt J, Bolla M, Joniau S, van der Kwast T, et al. EAU Guidelines on Prostate Cancer. Part 1: Screening, Diagnosis, and Local Treatment with Curative Intent—Update 2013. Eur Urol 2014 Jan;65(1):124-137. [CrossRef]
- InCites Journal Citation Report 2017. URL: https://incites.clarivate.com [accessed 2019-10-01]
- Dimensions. URL: https://app.dimensions.ai/discover [accessed 2019-07-01]
- Chang J, Desai N, Gosain A. Correlation Between Altmetric Score and Citations in Pediatric Surgery Core Journals. J Surg Res 2019 Nov;243:52-58. [CrossRef] [Medline]
|AAS: altmetric attention score|
|IF: impact factor|
Edited by G Eysenbach; submitted 25.07.19; peer-reviewed by F Kaliyadan, C Liebram; comments to author 14.09.19; revised version received 11.10.19; accepted 17.12.19; published 12.02.20
©Gregg Murray, Rebecca Hellen, James Ralph, Siona Ni Raghallaigh. Originally published in JMIR Dermatology (http://derma.jmir.org), 12.02.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Dermatology Research, is properly cited. The complete bibliographic information, a link to the original publication on http://derma.jmir.org, as well as this copyright and license information must be included.