Update: Communities/Hivemind
Hello Steemians! This is @roadscape, member of Team Steemit, and developer behind hivemind - the backend service which will soon introduce next-generation APIs and Communities for Steem. Now that it’s nearing completion, I would like to share a little information about how it works and what it will do.
More Than Communities
Hivemind (or “hive”) will power Communities on Steem, but that’s only one important use case. It’s designed to be a developer-friendly layer on top of the Steem blockchain that will enable any developer to access a database containing all the social information stored on the Steem blockchain without having to rely on minimalistic core blockchain APIs. As with all the software we release, hive is fully open source. It is our goal that everyone developing applications powered by Steem will want to take advantage of hive, so that a consistent user experience can be maintained across the growing ecosystem of Steem applications.
Hive is Developer Friendly
Currently, most Steem-powered applications rely exclusively on steemd full nodes as the backend for information like trending lists, account profile information, reputation scores, comments, follows, and so on. This model has certain strengths which have gotten us quite far, but developers have outgrown it and need a more powerful solution. Relying on steemd nodes prevents frontend developers from iterating rapidly, as (1) all the APIs are written in C++, and (2) any (and all) code changes must go through a rigorous review process because they are close to the critical consensus layer. Thus, it’s quite clear the concerns of application developers (flexibility/simplicity) and blockchain developers (security/stability) need to be separated.
Much of the information that app developers want to leverage is not consensus-critical; that is, it either does not belong in a blockchain node or it doesn’t need to be accessed directly from one. Instead, we can store all the relevant information in a traditional database and keep it in sync with the blockchain, while adding new custom layers of logic and features. This is exactly what hive does; and because it’s written in Python, developers can modify its behavior and APIs with ease... and experiment to their heart’s content.
Ensuring High Fidelity
Despite Hive being “non-consensus” it is important that the information it serves is accurate and up to date. The main challenge here is fork resolution; witness nodes handle this automatically with their in-memory undo database, but it’s not so easy for an outside observer. We could avoid forks entirely by following the last irreversible block, but this would result in a delay of around 45 seconds. Instead, hive (by default) trails behind by just 6 seconds, allowing us to avoid the most common case of harmless microforks. These are trivial for witness nodes but would require extra mechanisms to properly handle externally. In the rare case hive encounters a fork more than 2 blocks deep, we’ve implemented a basic recovery routine. Developers who wish to live on the wild side have the option to run hive with a 0-second delay; those who would rather play it ultra-safe can specify a delay of up to 5 minutes (100 blocks).
Hive is Specialized
Hive focuses on posts, relationships, social actions, custom operations, and derived states.
- discussions: by blog, trending, hot, created, etc.
- communities: mod roles/actions, members, feeds (in 1.5; spec)
- accounts: normalized profile data, reputation
- feeds: follows/unfollows and resteems/unresteems (yes, unresteems!)
Hive does not track most blockchain operations. For anything to do with wallets, orders, escrow, keys, recovery, or account history, you’ll want to query SBDS or steemd.
Hive is Creativity
Hive can be leveraged to create new features. Here are some that will be possible with hive:
- reactions, bookmarks
- comment on resteems
- indexing of custom profile data
- reorganizing of old posts (categorize, filter, hide/show)
- voting/polls (democratic or burn/send to vote)
- modlists: (e.g. spam, abuse, bad taste)
- crowdsourced metadata
- mentions indexing
- full-text search
- rich statistics
- follow lists
- bot tracking
- mini-games
- community bots
- and much more
It has a lot of potential I’m excited to see leveraged. That’s why I have dedicated so much time to making sure it works great: because hive will lay the foundation for a much richer and more social experience on the Steem blockchain, whether you’re using steemit.com or any other site which chooses to take advantage of this new open source layer.
Development Status: ALPHA
Hive 1.0: Core
The 1.0 release contains a MVP API which replicates all the relevant bits of get_state and get_discussions_* needed to support condenser. This is really quite boring, but it’s crucial to get steemit.com running against hive so it can be tested for load, reliability, and compatibility. The immediate goal is to take over certain steemd responsibilities while establishing the foundation for next-gen APIs.
The core indexer and API service are nearing completion. Because this is critical infrastructure, we need to ensure reliability and consistency. Currently, work is focused on launching the production environment and tuning performance. Then we begin integration and API testing. Once stable, condenser (powering steemit.com) will be switched over to the new hive endpoints; if everything is integrated properly, users will not notice a thing. It’s akin to upgrading an airplane engine mid-flight while making sure there are no hiccups and all controls work as usual.
Currently, we’re looking at the results of our first production reindex. As you can see in the chart below, we’ve had two large surges of activity on our blockchain. Recent blocks take up to 0.03 seconds to process. This may not sound like a lot, but it adds up.
If there exists a blockchain with anywhere near this quantity of rich and open social data, I’d be surprised (but do let me know). For those keeping score: Steem witnesses have produced over 20,900,000 blocks containing more than 285,000,000 transactions so far!
Hive’s full potential will not be unlocked until the new, modern APIs are finalized. We’ll post another update once we’re ready for beta testers and API feedback.
Hive 1.5: Communities
With the hard part out of the way, the fun begins with hive 1.5. Communities bring a whole new form of interaction to the Steem blockchain, and we are excited to see the ecosystem evolve once users can collaborate easily with like-minded individuals, curate topics of shared interest, and truly organize. We anticipate that the Steem blockchain will become the place to be to discuss everything and anything under the sun.
We know that everyone is as excited as us for Communities to finally launch. We still have some work to do before they are 100% ready, but we are very close to having 1.0 complete. We will continue to keep you up to date with our progress.
Stay tuned!
@roadscape, Team Steemit
Thank you for this great update @roadscape! I can't tell you how excited I am for all of this to be available and how appreciative I am of the hard work you're doing to make it happen!
Please let me know if there is anything I can do to help.
It’s my intention that hive creates the infrastructure which helps ALL contribute much more effectively. It’s amazing to see how far we’ve come with such primitive tools. But it’s time for a major upgrade.
Hive core gives developers a flexible layer which allows them to create more powerful tools and interfaces.
Hive communities give everyone the ability to ORGANIZE around a set of values, and the power to do so effectively. It also introduces a new dynamic of influence which does not have anything to do with SP. Minnows can mute whales. Unlike other platforms the moderation log will be fully transparent.
Such a system will be far more effective than anything we have today for connecting creators and curators. Curators will be attracted to communities which are well-organized: focused, high-quality, low-noise.
Easier to curate good content = easier to get paid for good content. Curators influence the flow of rewards, but their tools are crude. Now the power will be in your hands.
I truly look forward to the future and the coming technology you describe. The current environment just doesn't end up being sufficient to attract and maintain committment from excellent content creators in each niche field. There are so many users and groups out there who are dedicated to keeping the quality of content on steemit as good as it possibly can be. Attracting top talent requires the face of the platform they arrive to see to be an extremely appealing one. One in which they can see themselves having success and building a strong following.
Better connecting creators and curators will (hopefully) go a long way to aiding in the building and development the of this environment.
I am, excited for what you all have and continue to develop. It sounds like exactly what this platform needs to make the jump to the next level.
The fact that you wrote these four words in bold characters, gives us hope.
Ask any of the (established) creators and curators in this network what's the most important thing to them in the process of content production and evaluation, and I bet at least 90% will reply: quality.
You can either build a blockchain where value is defined through speculation or a blockchain where value is backed by a product in top quality. An outstanding product is also the no. 1 driver to a stable user retention.
Looking forward!
That's the short of it ;)
I can only join my voice to those of all Steemians focusing on quality content on top of anything else. I am so happy to read your words and the fact that content still matters for the people up there.
I am as well very excited to try all new functionalities as soon as they will appear!
@roadscape, this all looks good and sounds promising, and.... well, I'm with @surfermarly in saying that "connecting creators and curators" sounds great!
The reason that makes me happy is that it has often seemed like almost everything that comes from STINC is about developers and apps and SMTs... which is all good and fine, but there's a large population of us who live down here at what you might call the "retail level" of Steemit.
Our concern isn't what we can develop but how we can interact with the site, with the content, and with each other. The biggest challenge we face is that content discovery is difficult and often clumsy. We have few fundamental social content tools to make the curation process easier. Even on Farcebook and twatter, you can (for example) sort the people you follow into "groups" from which you can create custom feeds.
I feel hopeful that hivemind/communities will allow for better categorization of content... and NOT ONLY for those who speak fluent C++. php or whatever technical underpinnings run this place. We need tools on Steemit that work along the lines of "explain this to me like I'm a five-year old."
Yes, it IS amazing how far we've come with "primitive tools!" (Thank God I still remember how to speak HTML!) and kudos for that. On the other hand, to gain ANY sort of mass market following, this needs to become a slickly coded, user-friendly, intuitive user interface.
I hope that's at least PART of what's on the way!
I couldn't have said this any better @denmarkguy!
What you call 'retail level' is the majority of users :-)
I've done all my blog posts in HTML, too, since the editor would've never allowed me to professionally design my articles. That's completely fine - for you and me and a couple of more people. But it's a barrier for the masses.
The average user needs usability, guidance and tools to easily bring their content on the blockchain. They also don't want to search content, they want it to be offered.
But hey, we've been giving the same advices for months. In Germand I'd say now:
:-)
Absolutely.
Curators are the future of community building and if this happens according to your comment, then steem will not only be the first of it's kind, it will be the best. The thought of the ability of any empowered curator to reward good content gives inexplicable joy.
Am sure a lot of people are looking forward to this, thank you! Quality is still an integral part of this platform, this is great news.
Kudos!!
Great to read about the quality focus and progress in general @roadscape. I was owning and moderating communities in other social networks that paid in the past and it was a game changer - positive as well as negative - hence i was so curious from the first announcement to hear more details on how that will work from the tech side and group / community structure - ownership / moderators - separate accounts to be created or not. Think I annoyed @andrarchy a lot with bombarding questions long ago. I can't wait to get that live soon but i learned to be patient - keep them coming!
It is so awesome to hear that the encouragement of quality content is part of the focus of these upcoming additions. I love that this will make it easier for curators to find and encourage the creation of engaging content.
Now this here is something that gets you excited and hopeful. Creators and Curators getting Connected is the way communities should function, and no other way.
When influence comes based on how well we are and not what amount of SP we wield, the system will get the respects it deserves and the users the reward.
Thanks @roadscape for the time put into this project.
that's indeed the way to go. Communities should take the responsibility to curate within their own communities thus strengthening the bond within. So looking forward to this to launch !
@roadscape I guess the community has spoken. :) Thank you for all the great work and the hours spent and the care you've put on this.
It’s amazing to see how far we’ve come with such primitive tools.I felt the same when I first came across steem around nine months ago. Though there were many tools created by the community they were not easily accessible to the new entrants on Steem blockchain. People generally join the platform thinking it is like any other social media/blogging platforms. But when they see that there is a learning curve that they were not aware of earlier, many people hang their boots. So it is important to make sure that these tools reach the hands of the right people who are in need of them.From my experience in running a welcome bot @steemladder I have realised that just directing the people to the Steem Blue Paper has helped them a lot and they are better aware of the platform, difference between Steem/Steemit and how to navigate the platform. Once the dev tools get better we will have more people from community addressing these issues and onboarding of new users should become much smoother. All the best.
On a side note we are currently working on a platform that will enable running publications like Medium currently does. People can start a publication by creating a separate account for the publication. Then users can create posts as a part of that publication(the posts will be published by publication steem account) and the rewards will be shared with authors based on the pre-defined agreement. Do you think it is better to go ahead with the development or is it better to wait for the hive to be released. Also would love if you can give some suggestions so that it will be easier for us switch from the current endpoints to hive based API endpoints. If we have some insights we can prepared :P
The first version of the hive API will offer the same endpoints, so if you wanted to start now it's an easy switch. However the v2 APIs will be more powerful; they will return a lot less data but the data that is returned will be much more useful.
Thanks for the clarification @roadscape. All the best for the release.
"Hive communities give everyone the ability to ORGANIZE around a set of values, and the power to do so effectively. It also introduces a new dynamic of influence which does not have anything to do with SP. Minnows can mute whales. " Can you elaborate more?
Communities allow groups to form and work together to curate content they care about, to their standards. We of course have this to some extent already, except it's not possible to "claim" a tag as yours in any official capacity, nor is there the ability to brand your community on-site, or to appoint moderators. Currently SP is the only moderation system; hive communities introduces more powerful options and the ability to OWN your collective space.
Wubalubadub dub...... Rick
were goin to bendigo
Yeah i feel in the same way. All depends on thinking that you can do every think and this guy is going forward.
thanks for creating a skeleton that makes steemit a better platform, i hope you guys might wanna solve the problem in traffic network, because sometimes i'm worried about it when it happens. :)
:/ I love that communities are coming to steemit and some of the features sound good too, but I would rather see communities integrated as tenants on the steemit platform.
I would like more support for community driven tools. I see support for bots, but really? Bots? There's so much more that has been made. This feels more like a monolith than a microservice. It would be nice if there were hooks that allowed users to link applications by tenant.
I'd like to see identity management instead of normalized profile data.
I dunno. I like the features, but I feel like I'm shopping at goodwill instead of macy's. Like I'm being teased by what this could be in another year.
Here's the thing: any API tool that could be used to make something interesting can also be used by a bot. That's how we got all the bots we have. Any tool which can be used for insight and interaction can be used for automation.
If you want to remove the ability to automate, you have to remove the ability to interact. You don't want that. That is exactly the opposite of what you want.
Personally, my disappointment with this is that it is intended to only cover some of the social network side of the database and not, explicitly, the rest of the available public information on the blockchain which is specifically about wallets and transactions and money. That's going to be a real problem for people who are currently doing things like research into the interaction between the social network and fiscal/financial transactions on the blockchain.
And before you go and look at some of my recent work, yes, that means some forms of automated detection of bot-like behavior in order to make it easier for people to filter out those accounts.
I'm curious as to what the API for content-access is going to look like for building new discovery systems for content off the top of it. It looks like Hive will be a great boon for that kind of research. I'm looking forward to that part and of the fact that it should be easily accessible in Python.
I'm a little discontent that I will continue to have to go through a third party database provider to continue to have access to some of the most important data on the blockchain. That is more than a little annoying.
It is, however, good to see any kind of real development going on. Finally.
Hive has a tight focus on social and ignores financial only because they are two different use cases with quite different performance/integrity requirements. You may be interested in SBDS which is concerned with all raw operations and account histories.
Hi @roadscape I was wondering do you know why steemd is showing duplicate data entry for all actions on the blockchain?
You got a 2.44% upvote from @emperorofnaps courtesy of @uknowjpbitcoin!
Want to promote your posts too? Send 0.05+ SBD or STEEM to @emperorofnaps to receive a share of a full upvote every 2.4 hours...Then go relax and take a nap!
I'm not really opposed to bots or automation. Bots are IMHO intended for convenience. I don't want to make this political though. There's lots of software out there, and for better or worse, it comes down to people.
I just know that because bots are automated that they can only give superficial interactions with data, users, etc... Basically, it's software that's...not a high priority for a lot of people. We have a large set of data and potential to build really meaningful software from it.
1000% agree with you though. I think we all were hoping for more and we get treated like kids in a toys-r-us.
People oversimplify the issue.
The question is not "bots or not bots." The question is "what are you doing?"
In particular, because of the near criminally under-featured nature of Steemit as a social media platform, bots are almost a necessity to function within the context of its operations. I literally would not have stayed on this platform without Ginabot, which does a basic set of functions that you should expect a social network to do for itself. Mentions, keyword tracking, and notifications.
Ginabot is a bot.
But then we have automated voting trains and content harvesters and auto-reply systems and general spammers and mutual voting circles and all of the other horrible things which make living on the steem blockchain a generally miserable existence for a creator. All of the things that subvert the stated intent of Steemit as a platform (albeit exactly in line with the mechanisms of reward as designed).
We really need to start focusing on software behaviors that have negative repercussions of one sort or another, and be willing to talk about the trade-offs that come along with those activities – but that's not how discussion happens here. Either it's a crusade or everything is hunky-dory; there's no room for anything in between.
Given that Toys-R-Us has just declared bankruptcy, and a well-earned one at that, your illustration is both amusing and pointed.
We can actually have quite deep interactions with data and users in an automated way. In a real sense, that's exactly what Communities are; software constructs which provide content direction for a self-selected group of individuals. But that's a useful function, and one that is directly connected to the nature of the content in question.
A lot of the system as it stands is orthogonal to the issue of content. That's a real problem. It provides a lot of opportunity for analysis and observation of human behavior in a very synthetic environment, but as the basis for a social network which incentivizes behaviors which are prosocial, it's a real issue.
But now I'm reminded of all the Geoffrey the Giraffe fursuits out there just waiting to be repurposed by furry cosplayers and I'm kind of excited.
I really must quote you here:
a) "the near criminally under-featured nature of Steemit" and
b) "all of the other horrible things which make living on the steem blockchain a generally miserable existence for a creator"
c) All of the things that subvert the stated intent of Steemit as a platform (albeit exactly in line with the mechanisms of reward as designed)
d) Communities are; software constructs which provide content direction for a self-selected group of individuals
Very Deep, thanks.
I am excited to see the communities ready!
That'a what I'm wondering too.. on where that data lives.
It appears that it will NOT be part of the blockchain.
How can they boast that they have a superior blockchian when their solutions is to add layers where the bulk of the data will live. Databases which ARE NOT decentralized and worse, the features which separate a blockchain from a regular system WON'T be part of that layer...
I don't see this as a good thing.
Fragmentation DOES NOT make a system superior.
Hive processes only the data on the blockchain, nothing proprietary. To "talk" to hive you write to the blockchain.
What does this mean then:
and:
That's what I refer to. How does this equal decentralized.
I understand what HIVE will be. I want to know what the entire system will look like once all these layers are implemented?
If it's all fragmented how can it be considered secure and blockchain based if more and more of the data is going to exist on regular systems?
It just means hive specializes in certain types of data for specific applications, and providing convenient APIs that wouldn't make sense as part of the steemd consensus node.
There's really not all that many layers here, and it makes for a more agile environment. If there was one daemon that did everything, nobody would be able to run it. So now we can have more lightweight engines talking to each other. You could run steemd+hive yourself and it will be less painful than a full steemd node while offering more power. Or you could run hive and sync off a public steemd node.
Because more users can run more lightweight services, I see this as increasing security and anti-fragility. The blockchain remains the source of truth, I don't see the issue with storing copies of blockchain data on "regular systems" for easy access.
Good to hear. Looking forward to what all this will look like and if it can alleviate some of the current issues by helping to build the real community this place could be. Hopefully it will encourage more engagement and manual curation again.
Some beautiful, long clamored for features in here. The way you describe it, it almost sounds like everything with steemit.com will turn out...just fine.
Soon.
Hopefully the steemit core of quality content producing people can hold themselves together that long against the barrage of negativity that seems to be choking up my feed lately. (Anti bots, anti whale, anti bernie, anti self-vote, etc).
Just...gotta...hang...in there.
Thanks for taking time out of your busy schedule to update us muddy folks in the trenches with this progress report.
Though, some of that negativity is warranted, and a bit funny to hear you, with your username of all people, saying it, heh.
I mean, on this platform, it's up to us to fix what we find to be an issue. That starts with identifying it and convincing others of it. I think that the bot situation is literally killing bots. But it won't get better if we ignore it. It's only gotten worse month over month this year.
And while the stuff mentioned in this post is great, it doesn't undo all the other stuff that needs fixing. That negativity is not actually negativity for its own sake, but a call to action.
This literally made me more excited about the platform! Thanks @roadscape for being an amazing and inspiring person. Thanks for clearing doubts! :)
This is hopefully going to make and filtering a LOT easier! I'm not a developer, so I can't directly build on it or review it, but I'm VERY excited to see what the community is going to do with these powerful new tools!
The high level support for a better user experience is something i can relate to and i'm excited to see it coming for Steemit! Currently i'm using steemit.com most of the time but there is a natural urge for things like unresteem.. if you think "i made a mistake, please undo!" and i think there are many other things which developers can now provide more easily to handle the big data created by millions of Steemians soon. Maybe it will support also things like:
..to make Steemit to a more natural extension of daily life.
Cool suggestions!
I've been waiting for Communities for more than a year now. For me, it's an essential piece of the social networking puzzle that'll make Steem usable. I've been following progress on GitHub. Look forward to Hive 1.5. Thanks for all the hard work over the many months.
I'm also quite intrigued by the stuff in 2.0+ - gamification, rep score replacement etc. Hope to see those in 2019.
Unless I'm mistaken, it seems like community names will be directly tied to Steem accounts. Sadly, a lot of high value, universally popular community names are owned by account name squatters. It'd be a shame to have stuff like "science", "movies", "gaming", or "france" run by random abusers. Do we have a solution for this?
👍communities are absolutely essential and needed ASAP. As I mentioned in another comment, it’s amazing to see how far we’ve come with such primitive tools. Can only imagine where the next generation might take us..
Wrt names, it's a concern but there's a lot of plain words on other platforms which are reserved or closed off. A lot of the best content is niche to some extent, with names reflecting that. If it's truly an issue we can implement a 2nd layer namespace.
I would be very interested in knowing this too. If accounts are the only way to form a community, how will we also interact? Does that mean one account for a community and one account for say, posting and commenting? Anyway, I appreciate you @liberosist asking the question.
Let a MILLION community account names bloom?
I currently buy from blocktrades..