How to overhaul the broken review system

TLDR: After briefly outlining the problems with the current review ecosystem, I detail in-depth a research & review platform which addresses these points and seems plausible to me. I further illustrate 4 imaginary researchers to better understand what this would mean in practice and try answering some additional questions the reader might have, like how this would actually be implemented.

I don't know a single person who would argue that the current AI research ecosystem is working out. Conferences are filled to the brim with meaningless work, AI generated slop is all over the place, hardly anything is fully reproducible, reviewers have no clue what they are talking about, authors have no clue what they are talking about, and the list goes on.

There are many things I believe need to be fixed, ranked by their importance:

• People must take responsibility for their own work. This includes facing meaningful backlash when producing bad or simply meaningless work - and the same applies to reviews!
• People must be incentivized to favor quality over quantity.
• The rating of a work should be weighted towards the judgment of people who have already shown their own expertise (and keep displaying it in recent works)
• Even though ML lends itself extremely well to reproducibility, as opposed to other sciences like Biology, the current review process never tries verifying any claims. Especially now, where LLMs are generally at a point that they can reproduce a lot of work, it seems absurd to not make use of this and try to automatically verify claims.
• ML is a very fast-paced space and yet there are only a few dates per year when one can submit his work for review. Good science is not done on the clock and yet when it is finished, it should quickly be accessible & judged.

I believe hardly any of these points are very controversial - so how is it that we are somehow still stuck with the current system? I think for the most part, there haven't been any reasonable attempts to drastically supersede the established system - the fact of the matter is that careers in academia are often still bound to these few special conferences and going against the flow isn't realistic.

Which is exactly why I think that the current situation might be perfect for it: if you believe there is a decent chance doom is coming, it doesn't take a hero to not care all that much about how many citations you have and which 4-letter combination you won in the conference lottery, but instead to focus on your work making an actual positive impact.

In the rest of this post I'll explain and illustrate a system that I believe addresses the previous points and is plausible to achieve.

Overview

Posts are scored using a universal scale from 1 to 10. This might roughly equate to:

1 - terrible, shouldn't be on this platform
3 - requires significant overhaul / uninteresting
5 - interesting but not significant
7 - significant for a small space
9 - significant for a big space

Alike to a review, people can leave a rating with an explanation, or instead simply discuss in the comments. A post's rating then becomes the weighted mean of all current reviews. This weight is composed of a self-confidence assessment of one's review and one's expertise. The expertise is quantified by how well one's own posts did.

A specific example of what this could look like:

Let be the score of a person's 'th post. Then their overall 'karma' is defined as

where g is

Visualized:

To get a feeling for what this would mean, consider these examples:

• a post with a rating of 4 would neither increase nor decrease someone's karma
• two posts with rating 3 and 5 would perfectly cancel out
• it takes 9 posts with rating 6 to achieve the same karma as 1 post with rating 8

Further, the score of a post, , can be quantified in the reviews and their weighting: Let be the score assigned by the 'th review, the confidence (a float between 0 and 1) and the karma of the user.

Note that users with negative karma cannot directly influence a post's ranking, though they could still leave a comment that may convince other users to change theirs.

Specifics

While the previous section gave a broad overview, there are still many specifics to lay out.

Reviews & Comments

It is safe to say that such a system would live and die depending on the quality of reviews and comments. Therefore, we should design an additional scoring system for reviews (and comments), which also feeds into a user's karma, to disincentivize crude reviews through the direct scrutiny of colleagues.

While a simple up-/downvote system might seem sufficient, it conflates 'many upvotes' with other factors than just 'quality of review' - one especially prominent example would be two equally good reviews receiving wildly different amount of upvotes simply as a consequence of the popularity of the underlying post.

[More reasons also later discussed in the FAQ]

Instead a global scale that is averaged among voters, like posts themselves, would be much preferable. To achieve an ease of user experience comparable to 'pressing upvote', I would show a small slider next to the review/comment, displaying some range between say -3 and 3. There users can quickly slide to the specific rating they envision and also see the distribution of other voters.

Interactive example of what this might look like:

It should be noted that in contrast with reviews of posts, assigning a score to a review/comment doesn't require writing anything. Total transparency is still granted but such an additional requirement would disincentivize people from scoring other people's reviews/comments to a detrimental point.

The final rating would once again become a weighted average taking into account confidence and capability. This would then be added to a user's total karma (using an exponential function similar to ) with some hyperparameters for balancing karma from posts, reviews and comments.

Endorsement

While this platform would start with some set of initial, (hopefully) high-quality users, we continuously want to integrate new promising researchers. Existing users endorsing new users in principle sounds like a good approach but the lack of responsibility associated with endorsing slop is critical (as seen by arxiv). As such, the first post of a user you endorsed will also be reflected in your own karma, with some coefficient, say . This would generally make the effect of scores between 3 and 5 still negligible while ensuring that scores meaningfully below leave a mark. At the same time, it incentivizes you to take some time looking over possible promising endorsements.

Delay

Conferences are overwhelmed with the quantity of work and reviews become more sloppy as a result. This is fatal for meaningful work which often requires reasonably engaging with the material before judgement. To not succumb to this fate, there should be a hard constraint on how much one is able to post.

Naturally we don't want to limit highly capable individuals yet cranks must suffer a significant delay. This puts near the idea that the delay you experience should be a function of your own karma, factoring in your own competence: one user may be allowed to post after 1 month again while another has to wait 6 months.

Karma Time Decay

People change over time, be it for better or worse. It's not reasonable to mainly estimate a person's capabilities today through a collection of posts/reviews from many years ago, even if they were groundbreaking or terrible. In a meritocratic system like the one described here, we must keep updating on more recent information.

As such it makes sense to generally factor in an exponential decay over time for karma acquired, say a half-life of 12 months. This is also to the benefit of younger researchers who don't need a long record of publications as long as their recent research delivers.

It might still make sense to display a non-decayed version as an estimator of someone's entire career but for all functional use cases, the decayed version should be used.

Feed

Not only arxiv but even accepted or oral papers are at this point one fish in a giant sea. Such a giant sea that it's absurd to imagine someone would be able to find all the actually meaningful work. While endorsements and delays should already shrink the amount of posts to a much more feasible number, even among these, prioritization is a necessity. Luckily we can trivially optimize the feed using the rating of a post (and an additional exploration term, boosting posts with few reviews).

This allows researchers to stay on top of the most important research even when time is short - also especially useful for new researcheres to quickly get an overview of the best papers in the last few months/years.

Co-Author Split

It's not only essential to highlight good research but also the good researchers behind them. In the current day and age with absurd metrics like citations and amount of accepted conference papers, putting your name on anything and everything is a no-brainer. Instead, on this platform, putting your name on bad/meaningless research can for once actually be to your detriment. Further, karma gets split among authors in some kind of agreed-upon ratio, elevating and rewarding capable individuals quickly rather than making them disappear in a sea of names.

Automated Reproducibility

Technical AI Safety lends itself very well to coding agents which automatically verify the claims made in a paper, especially because of the small amounts of compute normally required. This is deeply needed seeing how many (oral!) conference papers can't be reproduced, making them close to worthless. I'm fairly optimistic that some scaffold third party would offer this for free as PR, once this platform gains traction, as long as we keep the amount of papers under control.

It should be noted that this would be a feature I would worry about down the line, not directly in the first version.

Optimal Weighting

With the current weighting we just plug in karma as a proxy for competence. Obviously weighting competence makes sense, but to get a more grounded weighting, we need to distinguish 'can better evaluate a research direction' from 'create a good research direction himself'. Specifically I'll argue we should actually take the root of one's karma for the weighting.

I think it's fairly accurate to model research as the process of taking many difficult, sequential decisions that lead to some final state once some time period elapsed. When we make these decisions, we rely on our intuition: we apply our value function for each option, if you so will. People with more experience and talent have a better value function, granting them closer approximations to the real value. This allows them to speed up the process drastically: even if you are only 5% better at picking the 'right' option, the effects will compound over time and you will find yourself in drastically different and better research directions than someone less skilled.

So, even if we assume karma as modeled right now is a good proxy for how good a proposed piece of research is, the result of recursively applying one's value function, this doesn't imply to overly weigh high karma opinions when time is short and the compounding can't properly take place yet.

Instead, when maybe only a few units of 'research time' have passed, we would like to mostly weigh by the base instead, the 5% from before. Importantly, this isn't the full truth either: people also invest 'research time' in a general world model that they can then take advantage of. This amortizes when reviewing and explains why, even when a reviewer only spends a very limited amount of time, this compounding is still happening to some degree.

The bottomline is that the most reasonable choice seems to be taking a (small) root of a user's karma and using that as weighting instead, say square root.

Examples

I'll describe short vignettes of 4 imaginary researchers on this platform to illustrate these abstract numbers and concepts in action. I would advise not skipping this portion.

Austin

Austin is somewhat of an outlier among his colleagues: he has decided to mainly spend his time reviewing and criticizing the work of others rather than focusing on his own research. Austin is fairly well known in the community and highly-looked upon for his in-depth and nuanced reviews - they often land at the top of all reviews.

Prior to this system, a researcher specializing in reviews would have been impossible but nowadays institutions and grants in the field mainly focus on a researcher's karma, relying on the expertise of the community.

Austin has now been doing this for almost 18 months. He generally writes 2 reviews per week which get a score somewhere between 2 and 3 on the review scale from -3 to 3. This translates to roughly 3 karma per month. After 6 months he had 16 karma placing him slightly above the average and now 34 karma, confidently in the top 20% of users.

This also of course means his reviews now have a significant pull to them: the median user with 10 karma has a strength of roughly 3 while Austin is around a 6.

Brandon

Brandon is a 2nd year PhD student. He put out his first post 7 months ago after being endorsed by his mentor - he reproduced an important earlier paper and found some peculiar things. The community rated it around a 5.8, roughly 3 karma, which he was happy with. 3 months later he put out his next paper (as soon as the delay let him), doing an extensive study of some existing approaches. This time around, it was less well received - Austin, an esteemed reviewer on the platform, rated it a 4: 'TLDR: There's little substance and meaningful results that further research should or could pursue, even while the results are extensive and sound'. That hurt.

Brandon looked around the platform, quickly identifying the latest highly looked upon papers and reflecting on his own work. In retrospect, Austin and the other reviewers weren't wrong and so Brandon changed his strategy: instead of looking for things to publish, he looked for hard problems he couldn't solve and boy there were many.

He has made some progress, but it will still take some time before he's finished with a full paper - not that this is of great importance on this platform, it has long moved towards publicizing intermediate results if meaningful enough. And so he reported on his latest breakthrough yesterday - and it seems this time the community agrees there really is something of substance, assigning it a rating of 7.1 or 10 karma!

Craig

Craig loves obsessing over mind-bogglingly difficult open-ended questions. He very rarely publishes anything and is in general not the most active on the platform - but that also just isn't needed for his research and obviously there's nobody forcing him: can you imagine a world where Craig would be loaded with random reviews (which he knows himself would NOT turn out great as a result) and useless metrics like papers published - how absurd!

When he does publish, it normally hits like a meteor: His last post from 8 months ago on a novel alignment technique received a rating of 8.2 or 41 karma. Occasionally, when he really enjoys a paper, he does write a short review, but he accounts for the fact that this hardly is his expertise by lowering his confidence - other people respect him for this.

Danny

Danny's career on the platform was short: he was originally endorsed as a coauthor, even when his contribution-split was only 5%. So even though the work was decent, 6.4 rating or 5 karma, he had less than 1 karma. His unobjective reviews quickly turned that into -3 karma, not that his weighting of 0.5 could have done much damage in the first place.

But not all hope is lost: while his ability to review might have been lost with negative karma, Danny's comments have become more nuanced and now actually slowly gain him karma again. He will put out his first first-author post soon and even though some people remember him as a nuisance, their reviews will be up to the same scrutiny of other colleagues - Danny has a fair shot just like anybody else.

FAQ

I'll now answer some questions I think a reader might have at this point in time.

Isn't this just LessWrong, approximately?

While some details are definitely inspired by LW, there are many things that differ and LW (categorically) doesn't offer. One of the strongest contrasts is the rating system disentangling amount of readers, style and actual quality of work. Let me go into slightly more depth:

I often find myself thinking "Why does this LW post have so many upvotes?". When I think this, I don't mean the post is bad - in fact, often I'm considering upvoting it as well. Yet when I put it in relative terms, I remember so many other posts which had much more valuable ideas. Does this post really deserve 5x their karma? At this point, I'm even considering downvoting the post - even though in a vacuum I would never!

LessWrong generally seems like a very high quality form of social media to me. But in the end it's just that: optimizing for someone to 'enjoy' their next recommended post rather than the post's quality. Was it funny to read? Was it easy to read? Did you need a deep technical understanding of something? Does it have some fancy graphs?

It is okay and maybe even optimal for LW to have all these things implicitly factored into the rating of a post. But it is NOT something we can accept for a research & review platform: of course there are still some stylistic confounds that are unreasonable to remove - god knows how many papers I didn't read simply because they read like shit - but we should aim to keep these as an elementary requirement rather than their rating.

The scoring system resolves this problem for the most part: you might have enjoyed something but when push comes to shove, how impactful is the work really? You can't just give it a 7 because you laughed a few times, your review is up to the scrutiny of your colleagues: if you give a 7, you better have a good argument why it deserves one.

What about the Alignment Forum?

The alignment forum seems mostly like a failure from where I'm standing, at least if its intended goal is anything close to what I'm proposing. There's much great research that doesn't get elevated to the forum, which doesn't stop it from having a significant portion of posts that I don't believe deserve to be there. It mostly carries the same problems as LW, for example the up-/downvote system as discussed above.

That said, when I haven't checked LW for some days, quickly checking the alignment forum instead gives me a decent signal cutting through some noise. It's a gradually better, but categorically doomed, system.

Will people be sycophantic?

Transparency has many benefits but it risks people sugarcoating their opinions, especially in the case of accomplished authors, so as to not risk potential backlash or souring relations. This is in my opinion the strongest argument against transparency, even if I don't judge it strong enough to actually supersede my desire for people to take responsibility.

First it should be noticed that while this transparency causes this issue in the first place, it also protects users: baseless, little objective comments and reviews can be directly traced back to a real person whose karma will be penalized through his colleagues. Especially the global rating system makes it difficult to incorporate biases, be it conscious or subconscious: downvoting a post with 4 downvotes and 40 upvotes isn't weird, but giving a post, where ratings range from 4 to 6, a rating of 2 is difficult to hide - and you literally need to give reasons as to how you have come to this decision. Ratings of comments and reviews operate by the same system and have the same difficulties for unreasonable bias; even if lessened (not needing a written explanation), their effect on someone's karma is as well: to single handedly meaningfully influence it through ratings of comments, there is a need for consistent downvoting.

This consistency is traceable and, at the very least, suspicious - it's not difficult to come up with graph algorithms to find these kinds of weird structures (consistently rates someone's comments significantly below average). Note that these could be benign (for example, a fundamental disagreement that, downstream, makes works seem drastically different in their importance), yet it should be enough to further disincentivize users from letting prior emotions flow into their judgement.

Lastly, there are still some things we could consider to take advantage of double-blind reviews while not impacting responsibility:

• always blind the author's name of a post its first week, so that posts don't become a self-fulfilling prophecy
• add an option for anonymous reviews but with the caveat that the rating incurred will still be added to your karma. This makes it necessary to somewhat disguise everyone's karma (for example, updated once a week) so that backtracking through karma is also impossible

How realistic is it to actually pull this off?

As I previously stated, I believe we are in a unique situation as a research field: for any other field I would deem it close to impossible but this field is young enough, condensed and most importantly filled with individuals that aren't risk averse.

I believe if some initial group of 15-20 accomplished researchers finds together, this would already be enough to give the system a fair shot.

What would a trial look like?

Let's say we got those 20 accomplished researchers together and they roughly agreed on the specifics - what then?

The job of these 20 researchers is then NOT to publish the greatest research ever and somehow try making the platform a 'thing' like that - not only because this will inevitably not work out. Instead, it's mostly a mix of networking & culture: when a colleague or fellow of theirs has done some work they believe deserves to be on the platform, they endorse them. When posts get published, they take the time to write some high-quality reviews and set the culture as such. Hopefully this will allow the platform to grow up to around 100 good researchers at which it should be able to sustain itself, without these 20 initial researchers specifically going out of their way.

At this point it will mostly be up to the system as to whether it actually dominates - there might be some very high level moderation adjustments but otherwise, the recursion built into this system must prove itself. If it can do this and showcase consistently good quality of reviews and posts, this platform would naturally become more favored: in the beginning with thoughts along the lines of "can't hurt to also put it on here" and later hopefully "do i even want to submit to a conference?".

On the technical side, nothing I have described is particularly difficult to implement (with exception of the automated reproducibility check). There should be some good security since accounts are very important but that's pretty much it.

1. you wouldn't believe how often I have seen people asking for an endorsement on some random social media (say BlueDot Slack) with obvious slop (like painfully obvious after looking for 3 seconds) and somehow, after a few hours somebody endorses it anyway. I tried asking some endorsers why but always just got ghosted
2. the examples are assuming single author posts - i'm open to splitting this delay up in some way, see the later section on coauthor credit: one could imagine these ratios not only being used to split the karma but also the delay incurred
3. all of this gets even more complicated when we factor in that some of the karma is directly from reviews - there we would probably not want to take the square root? but to not overcomplicate, i'll leave it at this
4. accounting for time decay