Tower: A REST API implementation on the top of Hivemind

in #utopian-io6 years ago (edited)

The official STEEM public node, api.steemit.com started routing some of the endpoints (follow api + tags api) to a Hivemind instance. It looks like the current migration performs well.

I have been running a public Hivemind node for a while. Current endpoint list of Hivemind is limited, however, it has a powerful database where you can filter and audit blockchain actions.

Hivemind doesn't sync every type of operation into its internal database. But, you can access

  • accounts
  • posts/comments
  • relationships (follows, reblogs, etc.)

with a direct SQL query.

Introducing Tower

Tower is a solution to expose the Hivemind's database as REST API interface. You can have a look at the documentation at tower.emrebeyler.me, and the base URL for the API is: https://tower.emrebeyler.me/api/v1/.

Example Calls (/api/v1/accounts)

  • Get the accounts in Germany, ordered by highest reputation

/api/v1/accounts/?location__contains=Germany&ordering=-reputation

  • Get the most followed accounts (Top10)

/api/v1/accounts/?ordering=-followers&limit=10

  • List the accounts selected @inertia as their witness voting proxy

/api/v1/accounts/?proxy=inertia

  • Get an account detail

api/v1/accounts/emrebeyler/

Example Calls (/api/v1/post_cache)

/api/v1/post_cache/?author=emrebeyler&ordering=-payout

  • Find @inertia's first post in the chain

api/v1/post_cache/?author=inertia&ordering=created_at&limit=1

Example Calls (/api/v1/posts/)

  • Find @emrebeyler's posts where the community was dpoll.

api/v1/posts/?author=emrebeyler&community=poll

  • Find an author's deleted posts/comments

/api/v1/posts/?is_deleted=True&limit=10&author=author


These are just example queries. Make sure to check the documentation for filters and ordering options.

Limitations

Available filters and ordering choices may subject to change in the future. Also, keep in mind that this is an experimental service and depending of the costs, my tower.emrebeyler.me may become private.

Technical Details

Tower is a Python3.6 and Django application. It uses DRF to expose database models as a REST interface.

If you want to run a Tower instance, follow these steps.

Installation

$ git clone https://github.com/emre/tower.git
$ python3.6 -m venv tower-env
$ source tower-env/bin/activate
$ cd tower
$ pip install -r requirements.txt

Configuration

$ vim tower/local_settings.py

Add database information of your Hivemind:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'db_name',
        'USER': 'db_user',
        'PASSWORD': 'db_password',
        'HOST': 'host',
        'PORT': 'port',
    }
}

Running

For development:

$ python manage.py runserver

For production:

$ gunicorn tower.wsgi

Roadmap

  • More filtering options on posts and posts_cache tables. These tables are huge and don't have many indexes. If I come up with a fast solution, I plan to add more filtering options.

  • Additional view sets for relationships (follow, reblog, etc.) These are already handled on the hive's internal RPC server, not a priority but good to have.

Post Updates

Vote for me as a witness

I do my best to support the blockchain with my skills. If you like what I do, consider casting a vote on via Steemconnect or on steemit.com.

Sort:  

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.


Need help? Chat with us on Discord.

[utopian-moderator]

Thank you for your review, @helo! Keep up the good work!

ǝɹǝɥ sɐʍ ɹoʇɐɹnƆ pɐW ǝɥ┴

What are your storage needs currently? I would like to do something similar, started off doing it in MySQL but getting all the data means gigs and gigs of space is needed.

postgres=# SELECT pg_size_pretty( pg_database_size('hive') );
 pg_size_pretty
----------------
 181 GB
(1 row)

181 GB for hivemind. But beware, Hive doesn't store all the data of the blockchain.

Thanks for this, I see it's a Postgres db in the back then.

I am interested in only a subset of the data. Is it possible to filter posts where the parent id is empty (i.e. only get back posts, not comments)?

Going to read up a bit more.

I am encouraged by the possibilities of what Hivemind can bring to communities and appreciate your efforts in development. I have given you my witness votes to assist you in gaining influence to continue to work towards the sustainability of the protocol!

Posted using Partiko iOS

Thank you! Much appreciated.

i want make a group exchange upvote ..

this is fantastic, nice work. Is adding global properties (steem supply, pricing, block number ....) on the cards at all?

Thanks. Yes, there is a hive_state table stores the dynamic global properties. I will add with the next iteration.

fantastic. Where can I find a full list of tables or structure of the data?

Here is an ER diagram for the current database structure.

Full resolution

brilliant, thank you

I have no idea what you're talking about, but still you had me at Hivemind :-) Does that mean we're close to Communities?

i want make a group exchange upvote ......

Source
When you copy/paste or repeatedly type the same comments you could be mistaken for a bot.

More information:
The Art of Commenting
Comment Classifications

Lol, hivemind topic is hot nowadays. :) But we're not close to communities yet judging by the activity on the github repository.

Hehe you bet! #hivemind will become THAT trending tag on Steem shortly.

I am unable to read what's published on github, I mean i can read it but don't comprehend - so thanks for the translation here :)

Awesome work! This is amazing. I can't even imagine the types of things that will now be possible to develop thanks to this. This likely means that developers need to know less about the quirks of working with the blockchain and can treat it more like a standard web service.

Fantastic Work!

Django .. Postgresql .. REST .. Hivemind

Screen_Shot_2019-01-14_at_8.49.58_PM.png
(Yes, it still works :))

Excellent work! Please be aware we just made a huge batch of updates to bring hivemind out of alpha into beta: https://github.com/steemit/hivemind -- many bugs fixed and indexes tuned/added. The latest migrations might take a while.

Great updates @roadscape! and congratulations on the beta milestone.

Reviewing the changes while taking a backup on the primary database. Will try migrating in a cloned database first. :)

Upgraded to beta. 🎉

Hello @roadscape i did not know in what way you can help me on steemit either by steem delegation. I have been posting on steemit but having low upvotes on my post and my account is not growing. Out of the low steem i get, i try to rebid bot to upvote my post so as to get more visibility but still not i am not getting much visiblity to my post.

I would be happy if you can surely help.me out with some good amount of steem power to grow up my steem account on steemit and i would really he happy for that.

Thanks for your helping hand for me to grow on steemit

i want make a group exchange upvote ....

This story was recommended by Steeve to its users and upvoted by one or more of them.

Check @steeveapp to learn more about Steeve, an AI-powered Steem interface.

Coin Marketplace

STEEM 0.21
TRX 0.20
JST 0.034
BTC 98833.06
ETH 3391.57
USDT 1.00
SBD 3.08