Hello everyone! I have just recently started the Spring semester at college. I apologize for not posting in the fall when school got busy, but I have still been working towards my goal of learning about programming with Steem in Python, and decided I have done enough to make a post!

What I have done so far? and Steem Veggies

I began a course on programming artificial intelligence (and data science) in Python over the break, and am a little over half-way through. I still have to actually develop the model (once I learn about all of the different options, and experiment with which ones work the best), but what I have so far is most of the work I think. At this point, what I have developed is a way to download the data in a clean way, and store it in a database. The course went over a lot about how to clean data, and how to visualize data. What I've realized is you can't really just feed an AI data, but rather you have to intuitively understand the data (which visualizations help you do), and pick the best model based on the data you have.

Anyway, I have realized that I need to make some tools for working with the data I have, and that these tools could eventually be useful to other people. So I am prematurely calling the tools Steem Veggies (pun intended). My hope is it will eventually break into three different vegetables:

Steem Asparagus - For working with individual account data such as author rewards, curation rewards, change in reputation, etc

Steem Broccoli - For working with blockchain data (I don't think this tool will be extremely important towards curation, but it could be)

Steem Carrot - For working with curation data pertaining to individual articles. This is what I currently have developed (for my use at least). The visualizations that I am using further down in this article was made using that tool.

If you would have an interest in using these tools, please let me know, and I may actually try to develop them for public use, but for now they are just for my own use in building and improving an AI.

Setup

First of all, I began this project months ago by setting up a virtual machine for linux on my laptop, and downloading the Steem Python library there. That was a week long task just figuring out how to get Steem to work in python, and I wound up having to do a lot of error searches, and installing new packages to make the installation of Steem work. Unfortunately the Steem Python library is not well-maintained. However, that virtual machine on my little laptop was not feasible, and I found myself spending more time waiting for it to load, than I actually spent working. So over the break, I had to come up with another solution, and I used WSL on my PC at home (now at school) to download pycharm, and learned how to run GUI apps on WSL. I then had to go through the entire process of setting up for Steem again, but once I did, it turned out to be a fantastic solution. Unfortunately I have to use python 3.8 because there was a huge problem in 3.9 working with Steem. I cannot emphasize enough that that library is not well-maintained.

What data am I tracking?
From there, I continued with my work which had been done on the laptop so far. The project began with a file for downloading articles from the blockchain, and extracting the data. Originally, I was storing the articles in a text file, but eventually when I was trying to delete every version except the most recent of articles that are edited, I realized that the text file wouldn't work, and @remlaps said I should learn to work with a database. So there I found myself on a week long tangent learning how to setup a mysql database. It was not hard, but very time consuming because I ran into a lot of syntax errors simply because using mysql in python involves just putting the code in strings in the execute function. So far the data I am recording is as follows (CK stands for the cksum which is a function for generating an integer to represent a word):

• Title
• Link
• Title Word Count
• Body Word Count
• Primary Language
• Primary Language CK
• Spelling Errors
• Percent of body that is spelled wrong
• Created
• Edited (boolean of whether it is edited or not)
• Total number of Edits
• Tags (only the first 5)
• Tags CK (only the first 5)
• Number of Tags
• Author
• AuthorCK
• Author reputation
• Total Beneficiaries
• Beneficiaries (list of them)
• Beneficiaries CK
• Total Votes
• Total Comments
• Percent SBD (whether it is 100 percent powered up or not)
• Total Value
• Author Value
• Curation Value
• Beneficiary Value
• Historic Values (at 4 to 10 min, 15 min, 20 min, 25 min, 30 min, 1 hour, 12 hours, 1-6 days)
• Historic average vote value (at the same time frame)
• Historic Vote Counts (at the same time frame)
• Historic Median Voter (at the same time frame)
• Historic Median Voter CK (at the same time frame)
• Historic Top Voter (at the same time frame)
• Historic Top Voter CK (at the same time frame)
• Post ID
• Post block number

The biggest challenge I faced in downloading this data was tracking the spelling errors. Originally, I was detecting the language of each word, and listing the language as one of the post's languages if that language had a certain number of words in the post. I then spell checked every word in the body in each of the dictionaries for the languages the post was in, and listed it as an error if it wasn't in any of the dictionaries. The problem came that most of the articles with bigger word counts and multiple languages were taking a minute or more to process. Almost all of the articles were taking a couple of seconds.

Eventually, what I did to solve this is I split the body into sentences rather than words, and detected the language of each sentence. I then put all of that sentence's words into a list in a dictionary with the language code as the key, and then after all of the sentences had been processed, spell checked each of the words in the list against their language. It now processes almost all of the articles in under a second. The problem is now that I do not know how to spell check articles written in Korean. I would appreciate @steemchiller's (or anyone else from the Korean community's) advice on how to spell check Korean if it's possible.

Some Data Visualizations

I have downloaded the data from December 18th, 2021 to January 18th, 2022 (176,902 articles). Here are some visualizations from that data!

Conclusion and Requests

Thanks everyone for reading this! Feel free to suggest visualizations to add, and leave feedback on whether you would want access to a tool for visualizations like this. I will make more updates as I do more work on this project. Thanks again for reading this! @pennyforthoughts will be used to give out 25% of the liquid rewards for feedback!