Review Process on PLUTO; with some exemplary cases
This part of the document is to help readers understand the peer review process of PLUTO with exemplary cases, and some potential issues to be dealt in near future. PLT token is a cryptocurrency specifically issued to be used on PLUTO platform which can be traded with other cryptocurrencies or fiat currencies on exchanges. Numbers in bold-italic are the detailed specifications subject to change.
Stories for three cases will be given as examples, two successfully undergoing peer review process of PLUTO with/without revision and an exceptional case. The peer review on PLUTO is basically open and public, meaning anyone registered can submit reviews to any articles. Researchers are strongly motivated to give decent reviews whenever they feel confident to do, with the aid of reward system in PLT tokens and reinforcing system of reputation. The reputation of individual researchers works as a base of weighting their intents on the platform. Every record in the process is disclosed after certain conditions are met.
Ordinary Case
Here’s a researcher, Alice, who recently got tired of having repeated controversies with reviewers from traditional journals. While talking about this with her colleagues, one of them introduces PLUTO. Alice decides to submit her paper to PLUTO, does some research on requirements to submit to PLUTO, and feels astonished to find out that she doesn’t need to reformat and revise her paper to submit to PLUTO. She registers to PLUTO with some of her research profile right away, gets some promotional PLT tokens for newcomers, and uploads her paper with 10,000 PLTs as reward for reviewers. Half of this reward is distributed with an automated algorithm on PLUTO, and Alice will choose the reviewers to receive the other half. Her paper directly gets into the Blind Review Period, in which anyone registered in PLUTO with their identification verified can participate to submit a review for her paper.
After a few days, some existing researchers on PLUTO with similar research subjects find out Alice’s paper, freshly submitted for blind reviews. They skim through the keywords and abstract of the paper to find that it fits to their interest, and go through a full depth review of the paper.
- Bob, Chris, David, Emma, and Frank pay some PLTs as deposits to prevent abuse,
- Bob, who had 2 Reputation on PLUTO, gives 7 out of 10 for Alice’s paper,
- Chris, with 9 Reputation, deliberately gives 2/10, lower than what he believes,
- David, with 3 Reputation, recommends some revisions on her paper,
- Emma, with 7 Reputation, gives 8/10,
- Frank, with 5 Reputation, gives 7/10, and
- at the same time, they all mandatorily give detailed comments on her paper.
In this Blind Period, all reviewing records are blind except that written comments are visible to the author, i.e. Alice, and if someone recommended revision she will be notified the comment which recommended the revision.
As Frank submitted his review, the contract on Alice’s paper fulfills the requirement of [4 reviewers && total 20 Reputation] for Blind Review, and automatically comes into the transition to Public Review Period after 72 hours. This 72 hours period divides into two sub-periods.
- First 48 hours as buffer for potential incoming Blind Reviews, and
- remaining 24 hours for Alice to choose the reviews that greatly helped her.
- No more Blind Reviews can be submitted after this 72 hours.
Alice has selected the reviews from Bob and Emma as greatly helpful. Now Alice has two possible scenarios. She can decide to (i) revise her paper during a designated period, or (ii) wait for the Blind Period to end without revision.
Submission of Article & Initial Blind Reviews
(i) Revising the article
As Alice determines to make some changes to her paper, she also has to determine how much time is required to do so, and she asks one week for her revision. This extended one week is independent from the period for Blind Review, and its sole purpose is to let Alice revise her paper and to let the reviewers update their reviews if necessary. When she uploads the revised version of her paper within the following one week, all of the reviewers are notified about the revision. As David has recommended the revision, he should now give a quantitative score with comments, and the rest of the reviewers can either update their reviews or stick with the existing ones.
Alice revises her paper with additional 1 week
David gives 6/10 for the revised paper with some comments. The other reviewers decides not to update their reviews. 48 hours after Alice uploaded the revised paper, her paper gets into Public Period.
(ii) Rejecting to revise
Alice might conclude that David’s suggestions are not so major to revise her paper, and choose to wait for the Blind Period to end. Same as the case (i), Alice thinks reviews from Bob and Emma well evaluated her article. In this case, her paper gets into Public Period right after the 72 hours buffer.
Alice rejects to revise
Alice’s paper now has a Blind Score, the average of scores from the Blind Reviewers each weighted by a function of the reviewer’s Reputation.
Meanwhile, as the Blind Period ends, the reviewers are compensated their reviewing contributions with PLT tokens according to how close their score is to the Blind Score. Those who closely evaluated the article will get Reputations as well, while those in the peripherals can loose some of their Reputations.
- As Alice chose Bob and Emma as helpful in both cases, half of the PLT rewards are evenly distributed among them. (2,500 PLTs for each)
- The other 5,000 PLTs are distributed among all Blind Reviewers. David in case (i), and Bob & Frank in case (ii) are compensated with the most PLTs, as well as increasing their Reputations. Chris, in both cases, is given almost no PLT reward, and is penalized with his Reputation as well.
- None of them can set their scores to maximize own rewards, as the process in Blind Period is all blind except that written comments are visible to the author, i.e. Alice.
David gets the most compensation, while Chris is penalized
Now Alice’s article is in the Public Review Period. The reviews submitted in Blind Period are all disclosed. In the Public Period, researchers can put more reviews and make discussions over the article progressively. Scores given in Public Period are also aggregated into Public Score in similar way. Unlike the traditional journals where papers are either accepted and published or rejected, Alice’s paper is directly published on PLUTO, and researchers can access her article with Blind & Public Scores supporting as community-based evaluations.
A Case Failing Blind Requirement
Greg, who has actively used PLUTO recently, thinks that some debris data from his personal experiment deserves submission and reviews. He uploads the dataset with 5,000 PLTs. Two researchers show enough interest to give blind review for his dataset, but no more review has been given afterwards. 1 month after he submitted his dataset, the contract for his submission automatically asks him with two options.
- Extend the Blind Period for 1 month with optional addition of PLTs to attract more reviewers, or
- Retract the submission to get refunded.
He decides to retract, his submission(i.e. his dataset) switches to private, and he gets refunded. Greg still has received two reviews from the field. He can now further develop his data with these reviews.
Issues to be Addressed
• Normalization: A nominally same score might have different meanings depending on the reviewing patterns of individual reviewers. A 5 given by someone frequently giving from 2 to 6 means much more than the same score from one giving 4 to 8.
• Abuse Prevention: Malicious actions such as spamming scores over every submitted article with meaningless comments must be automatically dealt by the protocol of the platform. It is challenging that defining an abuse in peer review is subjective, and distinguishing abuses from normal reviews is a necessity. Those highly unlikely can still be dealt with platform governance, not with the automatic protocol.
•Distribution of Reputation: The distribution of user reputations or their weight functions, on PLUTO must be modeled in a way such that the one with greatest weight still cannot expect to have control over the public intent.
•Compensating Mechanism: We need a sound model to distribute the PLTs to reviewers and reinforce their reputations. Much like preventing the abuses, setting a criteria for deciding “correct” evaluation is a challenge.
All these issues are possibly easy to handle if we had an abundant data to refer to. Current journal system are mostly centralized, without transparently disclosed data available.
The review process and the considerations on its design are still immature. We’d really love to hear from everyone on how to improve the idea to innovate the scholarly communication system. Feedback is welcome in any of our channels, and of course this Medium publication.