3 days ~ Evaluating the curation effort of aicu

My curation bot @aicu has been running for a couple of days now, let's look how well it's doing.

Cold Hard Numbers

My bot has been watching the "new" category of steemit for three days. Minus some crashes at night due to overheating. I'd say that's roughly 48-55 hours of runtime. In this time it resteemed and voted on 46 posts which the bot categorised as exceptional content.

If you're curious about how a bot determines such a subjective concept: I trained a machine learning model on a list of ~9000 articles. The articles are annotated with "picked by a curation network" or "not picked". The features only contain features which would not bias the classifier towards patterns that are discouraged in the steemit whitepaper (only voting on trending posts, scalping rewards etc.). I excluded features like the number of voters, the voters themselves, comments, author name and the post reward. The model looks at the text itself (TF-IDF Model +POS Tagger) and various metrics from the post itself like HTML tags, the number of images, post readability, the main post category etc. I don't want to discover trending posts with this bot, but interesting authors.

The vote and resteem are cast after roughly 14 minutes. I'm thinking of reducing that time window, as that would give more reward to the author. Aside from that, if cheetah notices a post in the 14 minute time window, the bot won't pick the post.

In the beginning, I set the bot to vote and resteem a post if 85% of the trees in the forest are confident that the post is high quality. After running for a while, I noticed that this threshold produced a few too many matches, and I increased the limit to 90%. It's been running at this setting since today, and I'll keep monitoring it from there.

Anyway, onto the performance of the bot. I decided to monitor performance not by the curation reward, but by the number of posts picked by larger curation networks after my bot has picked a post.

46 posts in total, of those are: 10 % picked by Curie, 6.5% C² and 6.5% travelfeed along with some others. Not too bad for a first attempt. I'm curious how this performance will change with the new 90% threshold.

Onto further feature engineering

But first and foremost I need to improve the features in the model. I've been thinking about training a Convolutional Neural Network as a feature extractor on images to incorporate this feature into the model. But I'm a bit sceptic whether this feature would introduce further noise. An interesting feature might be switching the HTML part of the model over to a TF-IDF model based on html tags including class names. Currently, the model is quite good at picking posts with high-quality images, without looking at them. Just reading a post seems to be good enough for the start.