Test - Documentation - Get - About


Automatic academic review processing for praise and criticism in English

PeerJudge estimates the strength of praise and criticism in peer review reports on academic papers. PeerJudge reports two scores:

-1 (no criticism) to -5 (very strong criticism)

1 (no praise) to 5 (very high praise)

Why does it use two scores? Because referees often provide balanced reviews, with the positive points not necessarily cancelling out the negative ones.

PeerJudge can also report binary (positive/negative), trinary (positive/negative/neutral) and single scale (-4 to +4) results. PeerJudge was developed for English, but can be configured for other languages by changing its input files.

Quick Test:

Output: Dual, binary, trinary, scale

Get PeerJudge

PeerJudge is free. Please contact the author Mike Thelwall for the Java version, whether for academic research or commercial use. It is provided without liability or guarantees for any uses. Downloading PeerJudge and/or the configuration files signifies acceptance of these conditions. To use PeerJudge for research only (free), please email from your academic email address.


PeerJudge is similar to the Java version of the sister program SentiStrength - see the SentiStrength Java manual and Mac users' starting instructions (also helps in Linux probably). Most of the information on the SentiStrength website is also true for PeerJudge. PeerJudge should be run with the command line option negatedWordStrengthMultiplier 1.

About PeerJudge

PeerJudge was funded by Jisc and developed at the Unviersity of Wolverhampton. Itis describe and evaluated in a published journal article.

Thelwall, M., Papas, E., Nyakoojo, Z., Allen, L. & Weigert, V. (2020). Automatically detecting open academic review praise and criticism. Online Information Review, 44(5), 1057-1076. https://doi.org/10.1108/OIR-11-2019-0347

Support files

The various files with PeerJudge contain information used in the algorithm and may be customised.

Language customisation

PeerJudge can be adjusted for other languages by translating the term list SentimentLookupTable.txt and adding any missing praise and criticism terms. The scores for terms should be in the range 2 to 5 (praise) or -2 to -5 (criticism). A score of +1 or -1 means neutral: neutral terms are ignored. A training corpus in the new language is recommended to help adjust the term weight strengths.

The following files will also need to be translated or replaced with a local equivalent (see the extra instructions):

Negating words occurring after sentiment words (e.g., "I am happy not" is OK in German but not English) can be customised in PeerJudge. PeerJudge may need the utf8 option to read the input files, if in UTF8 rather than ASCII format (note that utf8 does not always work on ANSI text files so it should not be used as the default).

PeerJudge versions for other languages

Would you like to help? If you are a linguist or non-English native speaker then you might be able to help us to make a version for your language.

Please email m dot thelwall at wlv.ac.uk if you would like to help. This makes a good student project.

Domain customisation

PeerJudge can be adjusted for other domains (e.g., humanities reviews, poster reviews, individual academic field reviews) by adding new relevant words and sentiment strengths to the term list SentimentLookupTable.txt and adjusting any relevant existing term strengths.