PeerJudge

Automatic academic review processing for praise and criticism in English

PeerJudge estimates the strength of praise and criticism in peer review reports on academic papers. PeerJudge reports two scores:

-1 (no criticism) to -5 (very strong criticism)

1 (no praise) to 5 (very high praise)

Why does it use two scores? Because referees often provide balanced reviews, with the positive points not necessarily cancelling out the negative ones.

PeerJudge can also report binary (positive/negative), trinary (positive/negative/neutral) and single scale (-4 to +4) results. PeerJudge was developed for English, but can be configured for other languages by changing its input files.

Quick Test:

Enter review text:
This study analyses variations in the cost of fish at a market in Wolverhampton. It is a well written paper with flawed methods.

Output: Dual, binary, trinary, scale

Get PeerJudge

PeerJudge is free. Please contact the author Mike Thelwall for the Java version, whether for academic research or commercial use. It is provided without liability or guarantees for any uses. Downloading PeerJudge and/or the configuration files signifies acceptance of these conditions. To use PeerJudge for research only (free), please email from your academic email address.

Documentation

PeerJudge is similar to the Java version of the sister program SentiStrength - see the SentiStrength Java manual and Mac users' starting instructions (also helps in Linux probably). Most of the information on the SentiStrength website is also true for PeerJudge. PeerJudge should be run with the command line option negatedWordStrengthMultiplier 1.

About PeerJudge

PeerJudge was funded by Jisc and developed at the Unviersity of Wolverhampton. Itis describe and evaluated in a published journal article.

Thelwall, M., Papas, E., Nyakoojo, Z., Allen, L. & Weigert, V. (2020). Automatically detecting open academic review praise and criticism. Online Information Review, 44(5), 1057-1076. https://doi.org/10.1108/OIR-11-2019-0347

Support files

The various files with PeerJudge contain information used in the algorithm and may be customised.

The SentimentLookUpTable is just a list of praise or criticism terms, each with the word then a tab, then an integer 1 to 5 or -1 to -5. This can be edited and extended. Strengths of +1 and -1 have no effect on the program. There are some in the list, just to indicate that they words have been considered but not used. Each word can end with a wild card *.
The EmoticonLookUpTable is as above but for a list of emoticons.
EnglishWordList.txt is just a list of English words - it is used for the part of the algorithm that tries to correct words with non-standard spellings.
NegatingWordList.txt reverses the polarity of subsequent words -e.g., not happy is negative.
BoosterWordList.txt increases judgement intensity -e.g., very poor is more critical than poor.
SlangLookupTable.txt – replaces common slang with equivalent words or expressions
IdiomLookupTable.txt–overrides the judgement strength of the individual words in the phrase

Language customisation

PeerJudge can be adjusted for other languages by translating the term list SentimentLookupTable.txt and adding any missing praise and criticism terms. The scores for terms should be in the range 2 to 5 (praise) or -2 to -5 (criticism). A score of +1 or -1 means neutral: neutral terms are ignored. A training corpus in the new language is recommended to help adjust the term weight strengths.

The following files will also need to be translated or replaced with a local equivalent (see the extra instructions):

EmoticonLookupTable.txt - check the strengths are appropriate and add any common new national variations
SlangLookupTable.txt – replace with a list of common slang in the new language
EnglishWordList.txt – replace with a word list of correct spellings in the new language (many such lists are on the web, but this step is optional)
NegatingWordList.txt – translate/replace with a list of negating words in the new language
IdiomLookupTable.txt–replace with a list of common idioms in the new language
BoosterWordList.txt – translate/replace with a list of booster words in the new language – words that emphasise the strength of emotion in any subsequent words
QuestionWords.txt– translate/replace with a list of words in the new language that reliably indicate that a question is being asked

Negating words occurring after sentiment words (e.g., "I am happy not" is OK in German but not English) can be customised in PeerJudge. PeerJudge may need the utf8 option to read the input files, if in UTF8 rather than ASCII format (note that utf8 does not always work on ANSI text files so it should not be used as the default).

PeerJudge versions for other languages

Would you like to help? If you are a linguist or non-English native speaker then you might be able to help us to make a version for your language.

Please email m dot thelwall at wlv.ac.uk if you would like to help. This makes a good student project.

Domain customisation

PeerJudge can be adjusted for other domains (e.g., humanities reviews, poster reviews, individual academic field reviews) by adding new relevant words and sentiment strengths to the term list SentimentLookupTable.txt and adjusting any relevant existing term strengths.