Background: Alternative bibliometrics or altmetrics, is a measure of an academic article’s impact on social media outlets, which is quantified by the Altmetric Attention score (AAS). Given a lack of data for altmetric trends during the COVID-19 pandemic, we conducted a comprehensive, multivariable analysis of top dermatology manuscripts published during this time period.
Objective: We aim to assess (1) the relationship between traditional bibiliometrics and Altmetrics and (2) factors associated with high AAS.
Methods: All abstracted articles published in the top-5 (ranked by SCImago Journal Rankings) peer-reviewed dermatology journals published in 2021 were included in our study. We collected AAS as the dependent variable and categorical predictor variables included journal title, whether a conflict of interest existed, open access status, whether the article was related to COVID-19 or skin-of-color research, and the type of research (eg, clinical, basic science, review, etc). Numerical predictor variables consisted of the impact factor of journal, total citations, and number of authors. Multivariable linear or logistic regression models were used.
Results: The relationship between AAS and citation number was significant by multivariable analysis during the COVID-19 pandemic (P<.001). Numerous factors, including studies related to COVID-19, whether the article was open access, title of the journal, and journal impact factor were also independently related to higher AAS (P<.002).
Conclusions: Our results validate the use of altmetrics as a complement to traditional bibliometrics, especially in times of widespread scientific interest. Despite existing in a complex realm of bibliometrics, there are also discernable patterns associated with higher AAS. This is especially relevant in the era of growing technologic importance and utility to assess the impact of scientific works within the general public.
With rapid web-based communication, scientific literature is more widely disseminated now than ever . Traditional metrics (eg, citation number), however, do not account for the dissemination of scientific research through general public engagement. Altmetric Attention score (AAS) is based on both the quality and quantity of mentions in media sources not considered by traditional bibliometrics and is calculated by a point system, with each attention source being assigned a specific weight ( ) [ - ]. While traditional bibliometrics may take time to progress, AAS tracks the immediate spread of scholarly work and assesses dissemination shortly after publication [ ].
|Attention source||Weight||Modifier notes|
|News outlet||8||News outlets ranked by tiers. More popular news outlets have greater contributions (eg, NY times>2Minute Medicine)|
|Policy document||3||+3 per policy document mention|
|Patent||3||+3 per patent in “different” jurisdictions|
|Wikipedia||3||Mention in multiple pages does not increase the score more than single page mention|
|Peer review (Publons, Pubpeer)||1||—|
|0.25||Only a curated list of public Pages count|
|0.25-1.1||Score range based on following (retweets have a 0.85x modifier):|
There have been recent concerns related to inadequacies of solely relying on traditional bibliometrics to measure impact . Thus, bibliometric research has been increasingly incorporating altmetric parameters. For example, a novel artificial intelligence bibliometric search engine model emphasized considering altmetric parameters as a strength of the model [ ].
Fields including radiology, pathology, or orthopedic surgery, have shown strong relationships between AAS and traditional bibliometrics [- ], representing a case for altmetrics to complement traditional bibliometrics [ ]. In dermatology, however, evidence of this relationship is scarce [ ].
The COVID-19 pandemic held unique ramifications for scientific article dissemination, resembling a sudden assault on an area of previously unknown vulnerability . These novel circumstances made early identification of research outcomes crucial for guiding clinical decision-making, decreasing the effectiveness of traditional bibliometrics [ ].
Given the novel circumstances, we suspected that there may be unique AAS trends in dermatology publications during the COVID-19 pandemic. With the increasing usage of social media in disseminating dermatologic information  and a scarcity of data on AAS trends in dermatology during the pandemic, we sought to perform a comprehensive Altmetric analysis to assess the following during this time period:
- What is the nature of the relationship between AAS and citation number?
- What factors are associated with a higher AAS score?
All full-length, abstracted articles published in the top-5 peer-reviewed dermatology journals ranked by the SCImago Journal Rankings (SJR) from 2021 were included in our study. As a newer generation measure of journal impact, SJR has been deemed a more reliable measure of impact by accounting for factors such as dilutional citations (eg, editor self-citations) which can falsely inflate a journal’s impact factor (IF) .
For each journal, we reviewed every abstracted manuscript from January-December issues of 2021 and AAS scores were collected using the Altmetric Bookmarklet tool (Euan Adie) . Other article-specific parameters were gathered as independent variables. The categorical independent variables included journal title, whether a conflict of interest existed, open access status, whether the article was related to COVID-19 or skin-of-color research, and the type of research (eg, clinical, basic science, review, etc). Numerical independent variables consisted of IF of journal, total citations, and number of authors.
All articles were accessed in a 2-week window in June 2022 to avoid variability in AAS changes.
Univariable linear regression or Kruskal-Wallis testing was used, and significant predictors of AAS were incorporated into a multivariable linear regression or quasi-Poisson logistic regression model, respectively. Calculations were done in RStudio (version 2022.02.0; Joseph J. Allaire).
We analyzed 747 articles, with an average AAS of 24.2. Articles related to COVID-19 or skin-of-color research consisted of 3.6% (27/747) and 1.1% (8/747) of the total article population, respectively. Clinical research was the most popular article type (494/747, 66.1%;). In our multivariable logistic regression model, “JAMA dermatology” predicted higher AAS and “Journal of the European Academy of Dermatology and Venereology” predicted lower AAS (P<.001). The presence of conflicts of interest and open access articles predicted higher AAS (P<.001). COVID-19 research also predicted higher AAS (P<.001), despite accounting for only 4% (27/747) of total publications ( ). In our multivariable linear regression model, both citation number and IF independently correlated with AAS, accounting for 20% of the variation in AAS (P<.001; ).
|Count (%)||Average Altmetric Attention Score||95% CI||Univariable P Value||Multivariable P Value|
|British Journal of Dermatology (BJD)||179 (24.0)||29.5||12.2-46.8||.27||—|
|American Journal of Clinical Dermatology (AJCD)||68 (9.1)||9.26||5.29-13.2||.32||—|
|JAMA Dermatologyb||107 (14.3)||66||44.5-87.5||<.001||<.001|
|Journal of the American Academy of Dermatology (JAAD)b||195 (26.1)||19.9||10.8-28.9||<.001||.72|
|Journal of the European Academy of Dermatology and Venereology (JEADV)b||198 (26.5)||6.31||2.79-9.83||<.001||<.001|
|Conflicts of interestb|
|Yes||8 (1.1)||87.2||–21.12 to 195.6||.01||.41|
|Systematic Review||118 (15.8)||15.4||8.70-22.2||.58||—|
|Narrative Review||84 (11.2)||8.46||5.1-11.9||.06||—|
|Case report or series||14 (1.9)||93.9||–42.4 to 230.3||.11||—|
|Basic scienceb||31 (4.1)||5.55||3.3-7.8||.05||.15|
|Other||6 (0.8)||8.67||–7.6 to 25.0||.39||—|
bIndicates variables found significant by univariable Kruskal-Wallis testing that were incorporated into the multivariable logistic regression model.
The magnitude of the relationship between AAS and citation number is not well quantified , but our multivariable model (AAS=IF*3.43+Citation Number*2.08–25.24, R2=0.2, P<.001) quantifies the strength of this relationship.
Previous studies have analyzed the attributes of the top COVID-19 articles [, ], but none have directly compared the AAS scores of COVID-19–related versus non–COVID-19–related articles. We found that despite consisting of only 4% (27/747) of the manuscripts published, the average AAS for COVID-19–related articles was nearly 10-fold higher than non–COVID-19–related articles, which was significant by multivariable analysis (P<.001). The COVID-19 pandemic represents the first international, widespread assault on human health in the modern era, resulting in a frantic race to understand and combat the novel coronavirus by scientists and laypeople alike. Thus, it is natural that the popularity of any manuscript related to COVID-19 would be reflected in both traditional and alternative bibliometrics.
Outside of studies related to COVID-19, we found that AAS was associated with citation numbers during this period. In dermatology, a previous study did not find a significant association between AAS and citation number , suggesting that the scientific interest sparked by the COVID-19 pandemic potentially strengthened the relationship between alternative and traditional bibliometrics. Given this close relationship, our findings support the potential for use of AAS complimentary to traditional bibliometrics especially in eras of global interest in scientific works.
Web of Science (WoS) is a database distinct from Scopus, the main database used in SJR. We used the Scopus or SJR database to determine the top-5 dermatology journals because journal rankings by discipline in WoS are not publicly published. Additionally, Scopus or SJR includes a broader spectrum of journals and provides a faster citation analysis . Nonetheless, assessing the differences between altmetrics in top WoS versus Scopus or SJR rankings is an interesting direction for future research as both databases have unique attributes [ ].
There are multiple reasons for the use of AAS. For researchers, altmetrics may be a clue for an urgent topic to investigate. For community members, the use of altmetrics promotes a form of patient-centered care, where the viewpoints of the general public are considered by investigators through the AAS. Greater consideration of altmetrics by current researchers in their scholarly pursuits may result in increased engagement with the future generation of researchers more likely to be on social media channels considered by the altmetric scoring system, thus inspiring young or aspiring researchers to focus on research topics particularly salient to society.
The profiles of the authors that publish in journals with higher AAS were outside the scope of our study as we focused on specific articles rather than authors. However, we found that the presence of conflicts of interest was independently related to higher AAS. We suspect that authors that publish in journals with greater AAS may have more funding or affiliations, resulting in an increased likelihood of conflicts of interest. In fact, AAS patterns seem to emerge early in training, as dermatologists from medical schools with greater research funding had significantly higher total AAS in their early research papers than those who did not (67.9 vs 22.9 P<.001) . Other author-specific parameters, such as H-index may also be correlated to higher AAS publications.
Limitations of our analysis include the exclusion of nonabstracted articles, which may have different AAS patterns. It is also important to note that AAS should only complement and not replace other traditional metrics, as they are not a traditional marker of scientific importance and may further legitimize sensationalism if taken in isolation . Nonetheless, we outline a model that demonstrates a significant relationship between AAS and traditional bibliometrics showing that there are several independent predictors of AAS of dermatology articles published during the COVID-19 pandemic.
The field of bibliometrics is undoubtedly complex, with multiple databases and methods to assess impact, and external forces, such as that of the COVID-19 pandemic, that can affect bibliometric patterns. Altmetrics are emerging as a unique tool in bibliometrics with utility that is especially relevant with the accessibility of technology in the modern era. There are multiple reasons to consider the public engagement of scientific manuscripts through alternative bibliometrics, including understanding of topics that are currently influential within the status quo, a focus on patient-centered care, and the potential to inspire future generations of scientists. While our work clarifies relationships between altmetrics and manuscript-specific factors such as citation number, further research is required to assess the relationship between author-specific, rather than manuscript-specific predictors of higher AAS.
Conflicts of Interest
- Warren HR, Raison N, Dasgupta P. The rise of altmetrics. JAMA 2017;317(2):131-132 [CrossRef] [Medline]
- Trueger NS, Thoma B, Hsu CH, Sullivan D, Peters L, Lin M. The altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med 2015;66(5):549-553 [CrossRef] [Medline]
- Altmetric attention score modifiers. Altmetric. 2022. URL: https://help.altmetric.com/support/solutions/articles/6000234288-altmetric-attention-score-modifiers [accessed 2022-10-15]
- How is the altmetric attention score calculated? Altmetric. 2021. URL: https://help.altmetric.com/support/solutions/articles/6000233311-how-is-the-altmetric-attention-score-calculated- [accessed 2022-10-15]
- Singh HP. Alternative research bibliometrics: it's about quality and not quantity. Shoulder Elbow 2022;14(2):121-122 [https://europepmc.org/abstract/MED/35265176] [CrossRef] [Medline]
- Maatouk Y. AI-SPedia: a novel ontology to evaluate the impact of research in the field of artificial intelligence. PeerJ Comput Sci 2022;8:e1099 [https://europepmc.org/abstract/MED/37346315] [CrossRef] [Medline]
- Floyd AR, Wiley ZC, Boyd CJ, Roth CG. Examining the relationship between altmetric score and traditional bibliometrics in the pathology literature. J Pathol Inform 2021;12:8 [https://linkinghub.elsevier.com/retrieve/pii/S2153-3539(22)00130-4] [CrossRef] [Medline]
- García-Villar C. A critical review on altmetrics: can we measure the social impact factor? Insights Imaging 2021;12(1):92 [https://europepmc.org/abstract/MED/34215935] [CrossRef] [Medline]
- Richardson MA, Bernstein DN, Mesfin A. Manuscript characteristics associated with the altmetrics score and social media presence: an analysis of seven spine journals. Spine J 2021;21(4):548-554 [CrossRef] [Medline]
- Iglesias-Puzas Á, Conde-Taboada A, Aranegui-Arteaga B, López-Bran E. Factors associated with high altmetric attention score in dermatology research. Australas J Dermatol 2021;62(3):e380-e385 [CrossRef] [Medline]
- Boetto E, Fantini MP, Gangemi A, Golinelli D, Greco M, Nuzzolese AG, et al. Using altmetrics for detecting impactful research in quasi-zero-day time-windows: the case of COVID-19. Scientometrics 2021;126(2):1189-1215 [https://europepmc.org/abstract/MED/33424050] [CrossRef] [Medline]
- Szeto MD, Mamo A, Afrin A, Militello M, Barber C. Social media in dermatology and an overview of popular social media platforms. Curr Dermatol Rep 2021;10(4):97-104 [https://europepmc.org/abstract/MED/34692234] [CrossRef] [Medline]
- Falagas ME, Kouranos VD, Arencibia-Jorge R, Karageorgopoulos DE. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 2008;22(8):2623-2628 [https://faseb.onlinelibrary.wiley.com/doi/epdf/10.1096/fj.08-107938] [CrossRef] [Medline]
- Bookmarklet for researchers. Altmetric. URL: https://www.altmetric.com/products/free-tools/bookmarklet/ [accessed 2023-07-25]
- Uysal BB, Islamoglu MS, Koc S, Karadag M, Dokur M. Most notable 100 articles of COVID-19: an altmetric study based on bibliometric analysis. Ir J Med Sci 2021;190(4):1335-1341 [https://europepmc.org/abstract/MED/33459942] [CrossRef] [Medline]
- Tornberg H, Moezinia C, Wei C, Bernstein SA, Wei C, Al-Beyati R, et al. Assessment of the dissemination of COVID-19-related articles across social media: altmetrics study. JMIR Form Res 2023;7:e41388 [https://formative.jmir.org/2023//e41388/] [CrossRef] [Medline]
- Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J 2008;22(2):338-342 [https://faseb.onlinelibrary.wiley.com/doi/epdf/10.1096/fj.07-9492LSF] [CrossRef] [Medline]
- Huang A, Zhu H, Zhou K, Kirby RP, Dasari N, Calderara GA, et al. Social media impact of articles published by dermatology residents during medical school: cross-sectional study. JMIR Dermatol 2022;5(3):e39201 [https://derma.jmir.org/2022/3/e39201] [CrossRef]
|AAS: Altmetric Attention Score|
|IF: impact factor|
|SJR: SCImago Journal Ranking|
|WoS: Web of Science|
Edited by R Dellavalle; submitted 19.02.23; peer-reviewed by Q Chen, E Said-Hung; comments to author 15.05.23; revised version received 04.06.23; accepted 17.07.23; published 16.08.23Copyright
©Harrison Zhu, Vishnu Narayana, Kelvin Zhou, Anisha B Patel. Originally published in JMIR Dermatology (http://derma.jmir.org), 16.08.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Dermatology, is properly cited. The complete bibliographic information, a link to the original publication on http://derma.jmir.org, as well as this copyright and license information must be included.