Understanding Scholarly/Academic Research

Impact metrics measure the impact of content on a discipline and the community. All have strengths and limitations in both design and output. 

• help to develop a perspective on your research output and the research impact of fellow scholars.
• may reveal the importance of a body of research and the impact of an individual scholar and highlight a work's positive and negative impressions.
• DO NOT assess the quality of a work; rather, they reflect the impact based on the usage of a specific work on a field of study, a topic or a discipline. 

Impact metrics are helpful and heavily used but should be viewed critically and applied with caution due to many issues that include:

• New research, new journals, early career scholars, and non-English content are under-represented by traditional impact metrics., e.g. Citation Tracking and Journal Impact Factor. 
• Typically focused on quantity (amount of output), not necessarily quality of the work.
• Publishers (eg. Elsevier-Scopus-Citescore) may rank their journals higher than the competition.
• Authors can manipulate their metrics through self-citing.
• No measurement of scholarly, non-academic work Eg. EBP study in a hospital setting.
• No standardization of metrics within or across academic fields.
• Publication frequency differs by format (paper, electronic) and time to publication (newspapers, blogs, journals, books).  This may impact similar content being less cited despite the quality of the research.
• Negative reviews of works cited as poor examples can increase item metrics.

CITATION TRACKING is a traditional academic measure of an author's or specific work's impact in a discipline

Examples of Citation-Based Metrics

• Citation Count: total number of academic citations a specific publication or group of publications has received. 
• Citation Impact: average number of citations a specific author receives per publication.
• h-Index:  citation impact  + productivity = researcher consistency of impact 
• i10 Index:  number of publications (in a given group) cited at least ten times.

Journal Impact Factor (JIF) is another ranking process that judges the influence of journals (not articles or people) relative to one another through citation counts. Factors that are considered are publication competition and editorial standards. Publisher and institutional influence may impact the results. 

Alternative metrics reveal how content outside the academic milieu is viewed and may be ranked for impact.  

Data Sources:

• Media, e.g. newspapers, tv news, journal websites
• Social media and web content, e.g. SSRN, LinkedIn, Twitter, Dataverse, Github, Facebook, citation management system bookmarks 
• Downloads of full-text articles, software etc. e.g. through a public policy think tank
• Comments/cites/mentions on blogs, online forums, podcasts etc.
• Faculty websites, e.g. university websites
• Public Policy documents, e.g. Center for Disease Control

Measurements:

• Mentions/ Shares:  # times research is mentioned in media or social media 
• Views/ Downloads: # times an article, dataset, presentation, etc., has been accessed from a given website or database
• Ratings/ Reviews: personal or professional evaluations of  research output reported through traditional or social media (e.g. book reviews in journals, product ratings on Amazon, etc.)