Python Script to Save All Your Posts As MarkDown or Html Files

I always wanted to be able to save my Steem posts locally. After that better searching tools are available than the ones we have at the blockchain level.

I have only started poking around the development APIs for Steem, and this is the first script with a real purpose I've done in Python. On top of that, I'm also kind of new to Ubuntu. :)

If you are a dev and have been doing this for a while, you probably can write a more efficient script.

I wasn't looking for efficiency when I wrote it, I was interested to learn, and from there maybe others who are also Python beginners or haven't tried to code using Steem APIs. Hence the extensive comments.

Features and options:

• saves all your markdown posts as .md files
• saves all your raw HTML posts as .html files
• you can set a main sub-directory or sub-path in the current directory where the files will be placed
• posts will be placed in subdirectories based on the creation date (year-month) or primary tag - option to set at the beginning of the script
• you can save the posts for any account
• you can save resteemed posts as well or not
• you can add tags at the end of the post or not
• title is automatically added as H1 at the beginning of the post

I've tested the script on Python 3.7.4, but I believe it should work on earlier versions. Also the script is written for Linux/Ubuntu, for Windows you will need to adapt the parts of the script handling paths and creation of directories.

You will also need a good Markdown viewer/editor to see the saved files. I used Typora, but it looks like this will be a paid software when it exits beta version, so a good free alternative will be nice.

So, here's the Python script. Pay attention, settings are hard coded, you'll have to manually change them.

While I'm far from a Python or Steem dev expert, if you have questions let me know.

Feedback to improve from more experienced devs is welcomed as well. :)

import os
import sys
import json
from steem import Steem
s = Steem()

# script parameters
# =================

# author
author_name = 'testuser123'

# relative directory under which the posts will be saved (don't add a final "/"!)
main_save_dir = 'steem-posts-' + author_name

# structure of directories under which posts will be saved
# Options:
# primary-tag - posts are saved under their primary tag subdirectory
# year-month - posts are saved under the year-month of their creation date subdirectory
dir_struct_option = 'year-month'
print('Save posts by ' + dir_struct_option)

# bool flag to determine if tags are added at the end of the post or not
adding_tags_to_saved_post = True
print('Adding tags to the end of each post? ' + str(adding_tags_to_saved_post))

# bool flag to determine if to save resteemed posts of other authors as well
include_resteem_posts = False
print('Include resteemed posts? ' + str(include_resteem_posts))

# =====================
# end script parameters
#

#create main save directory (as a subdirectory or sub-path of the current directory)
try:
    os.makedirs(main_save_dir)
    print('Directory ' + main_save_dir + ' created in current directory ' + os.curdir)
except FileExistsError:
    print('Directory ' + main_save_dir + ' already exists in current directory ' + os.curdir)
except OSError:
    print('Directory ' + main_save_dir + ' couldn\'t be created in current directory ' + os.curdir)

#save current dir
cur_dir_saved = os.curdir

# loops through all the posts of the given author
# we break out of the loop after we reach the last post of the author
i = 1
while True:
    
    #retrieve current blog post info
    #theoretically we can retreieve more than one blog per call, in my tests anything more than 2 generated an error, so I prefered to take them one by one
    try:
        blogs = s.get_blog(author_name, i, 1)
    except Exception:
        print('Couldn\'t get blog #' + str(i) + '. Trying again. Ctrl+C to interrupt.')
        continue
    #is it empty? then we reached the end and we should break out of the loop
    if blogs == []: break

    #is it the author's post or a resteem?
    #if it's a resteem continue from the next iteration and resteems are not to be included
    if blogs[0]['comment']['author'] != author_name:
        if not include_resteem_posts:
            print('Post #' + str(i) + ' author is ' + blogs[0]['comment']['author'] + '. Skipping it.')
            i += 1
            continue
        else:
            print('Post #' + str(i) + ' author is ' + blogs[0]['comment']['author'] + '. Including it.')

    #choose the name of the subdir where to place the saved posts
    #(i.e. posts can be saved by primary-tag or date [year-month])
    if dir_struct_option == 'primary-tag':
        subdir_name = 'tags/' + blogs[0]['comment']['category']
    elif dir_struct_option == 'year-month':
        subdir_name = 'date/' + blogs[0]['comment']['created'][0:7]

    #attempt to create the subdir first
    if cur_dir_saved == '.':
        dir_name = main_save_dir + '/' + subdir_name
    elif cur_dir_saved == '/':
        dir_name = cur_dir_saved + main_save_dir + '/' + subdir_name
    else:
        dir_name = cur_dir_saved + '/' + main_save_dir + '/' + subdir_name

    #create the subdirectory/ies where we will place our files
    try:
        os.makedirs(dir_name)
        print('Directory ' + dir_name + ' created.')
    except FileExistsError:
        pass
    except OSError:
        print('Directory ' + dir_name + ' couldn\'t be created.')
        raise OSError

    #deserialize json_metadata
    json_metadata_str = blogs[0]['comment']['json_metadata']
    json_metadata_dict = json.loads(json_metadata_str)

    try:
        format = json_metadata_dict['format']
    except KeyError:
        print('Broken blog json before format key. Defaulting to "markdown+html".')
        format = 'markdown+html'

    #is the post markdown?
    if format == 'markdown+html' or format == 'markdown':
        #choose the filename as the blog post's permlink + ".md" extension
        filename = blogs[0]['comment']['permlink'] + '.md'
        
        if (adding_tags_to_saved_post):
            #get tags and create a string with them to add at the end of the post
            try:
                tags_str = '\n\n'
                for x in json_metadata_dict['tags']:
                    tags_str += '#' + x + ' '
            except KeyError:
                tags_str = ''
        else: tags_str = ''

        #get post body
        body = blogs[0]['comment']['body']

        #get post title
        title = blogs[0]['comment']['title']

        #format the body to also include title at the begining as H1 and tags (with #) at the end
        body_with_title_and_tags = '# ' + title + '\n\n' + body + tags_str
    #or is the post raw html?
    else:
        #choose the filename as the blog post's permlink + ".md" extension
        filename = blogs[0]['comment']['permlink'] + '.html'

        if (adding_tags_to_saved_post):
            #get tags and create a string with them to add at the end of the post
            try:
                tags_str = '\n\n'
                for x in json_metadata_dict['tags']:
                    tags_str += '<a id="' + x + '" href="#' + x + '">' + x + '</a> '
            except KeyError:
                tags_str = ''
        else: tags_str = ''

        #get post body
        body = blogs[0]['comment']['body']

        #get post title
        title = blogs[0]['comment']['title']

        #format the body to also include title at the begining as H1 and tags (with #) at the end
        body_with_title_and_tags = '<h1>' + title + '</h1>\n\n' + body + tags_str

    #write post to file (overwrite if exists)
    try:
        f = open(dir_name + '/' + filename, 'w')
        f.write(body_with_title_and_tags)
        f.close()
        print('Post #' + str(i) + ': ' + dir_name + '/' + filename + ' successfully saved.')
    except OSError:
        print('Something went wrong while attempting to write file ' + dir_name + '/' + filename)
        raise OSError

    i+=1

print('No (more) posts.')

Update: Edited the post because in the original there were some errors due to the copy-pasted code to html, which I haven't initially tested.