Programming Diary #33: SCA now provides tag-following

in Steem Dev16 hours ago (edited)

TL;DR: The SCA has been updated to include activity displays when Steem posts or comments are created under selected tags.

Summary

This post describes recent and coming program activity with my Steem-related Open Source projects. The main thrust of recent week has been an effort to activate tag-following in the Steem Conversation Accelerator (SCA). The SCA will also be the main focus of my efforts in the immediate future.

As with previous programming diary entries, the post also contains a "Reflections" section. In this post's reflections, I comment on the handling of accounts that are muted by condenser or by the Steem APIs; some quirks of post handling when using the database_api.get_comments call; and also the interesting effects that US politics and AI technologies may have on the global crypto-economy and also on Steem's ecosystem at this unique moment in history.

Background

You can find all four of my Open Source browser extension repos, here. They are: the Steem Curation Extension, the Steem Follower Checker, The Steem Conversation Accelerator (SCA), and SAM (The Steem Activity Monitor).

Please consider trying them out and letting me know how you make out. I also invite Steem developers to help make them better. It's a spare time hobby for me, so immense opportunities for improvement are definitely available (Caveat: As-of now, I have no plans for future SAM updates. That was just a toy project that started off as an AI prompting challenge.)

Here were my short-term goals three weeks ago, when I posted my last Programming Diary.

Priority-wise for the next interval, I will focus first on the SCA and second on the stealth mode project. I thought before that I might have tag following ready to release by mid-January, but now I'm thinking it'll be the end of January.

It turned out that I only found time to work on the SCA.

Activity Descriptions

During the past three weeks, I implemented a very basic tag-following mechanism that can be used to get notifications when Steem posts or comments are entered under selected tags. I think the entry form is basically unchanged from previous weeks (though I added options for 90 and 120 minute polling intervals). Here's what it looks like:

Tag list collapsedTag list expanded

I also worked on updating the display format for tag-following and added a clickable list of tags to the activity display in the section for followed accounts.

The current operation goes as follows:

  1. Popup HTML form and Javascript program: Accept config settings and save them in browser storage.
  2. Background JavaScript: Polls followed accounts and recent posts. Whenever activity is observed by a followed account, a browser notification is delivered (note that notifications are not yet implemented for followed tags, but they generally get swept in with the followed account notifications).
  3. ActivityList HTML and JavaScript: In a new browser tab, display post data from followed accounts (posts, comments, replies) and activity under followed tags in the post, itself, or in the reply thread below some root level post.
Here's what the browser tab looks like with tag-following added

Overall, I'm happy that the capability is available now, but there are many things that I want to improve - some that I've mentioned before and others that are new.

Next up

The main challenges with the SCA right now have to do with time. For my account (following about 860 users), a single poll takes about 15 minutes. This is much too long. Additionally, if I uninstall/reinstall the browser app, I need to reenter the followed tags, which can be a time consuming process. In addition to fixing these time-sinks, I also want to add additional notifications for conversations in which the observer account has participated and provide visibility for authors who make use of the blockchain's token burning services. Finally, I want to provide a more robust configuration capability to let the user turn on/off specific classes of notifications. (Much) Longer term, I may even want to merge the SCA, the Steem Curation Extension, and/or the Steem Follower Checker.

For the "stealth project" that I posted about last week, I think I'll need to push it back a year. I just haven't had time to focus much attention on it, and I want to make multiple improvements to the SCA before I do. The "stealth" project needs to have a May release date, so if I miss May/2025, that probably means May/2026 (hopefully).

Lastly, @cmp2020 and I still need to find some time to focus on polishing and bringing his Steem Curation Extension changes into the main branch.

SCA

Here are the planned updates for the SCA.

Revamp the account following mechanism

The hair-brained way that I implemented checks for activity by followed accounts was by using condenser_api.get_account_history to check the account histories for each of the followed accounts. This turned out to be very slow. Especially for witnesses and accounts that do a lot of voting.

To be fair, it wasn't really hair-brained because I didn't know of any alternatives at the time. But now that I've implemented tag following, I realized that I could use the same mechanism for followed accounts, too. So, it only seems bone-headed in hindsight. Tag following is making use of the database_api.list_comments API call, and that call should also be able to provide much of the required info for account following and for conversation following.

Another issue with the followed accounts check is that it constrains decentralization. Unfortunately, account following also depends on the follow_api.get_following API call, which is not supported by many of Steem's public API endpoints, so the browser extension is limited as to which APIs it can use. As-of now, I don't know any way to collect the follower list without that API call (or switching to SDS, which would also constrain decentralization). I don't think that has much impact on time lags, though, so it's not a priority.

Save/extend config settings

After tag following, in order to support version upgrades, the extension will need to be able to save its followed tag list in a more permanent way. So, I'll be adding a mechanism to export settings to a permanent location and import them at will.

Also, as I start providing the capability to follow more things, I want the browser owner to be able to opt in and out of each specific notification class. For example, maybe someone wants to get notifications on followed tags, but not from followed accounts. So, I'm planning to add that level of detail and control to the configuration popup.

Add conversation-following and Visibility as a Service (VAAS).

These have been on the roadmap from the time that I first implemented this browser extension. For conversation following, the idea is that if I participate in a conversation, I might want to get notified about comments/replies that are not direct replies to my own comment. For VAAS, the idea is to provide a blockchain based publicity service. People can obtain visibility for their posts by burning STEEM and/or SBD.

Display format

After switching data collection to use the database_api.list_comments API call, it will make sense to reorganize and consolidate the display format so that each specific data filter doesn't need to appear in its own section.

Reflections

Muted accounts

One of the things that testing with tag-following has shown me is that there are some number of accounts that are hidden at the condenser and/or API level, but continue to happily post away 5 or 10 or 20 times per day. From a development perspective, this raises the question of whether or not I want to display these accounts in the browser extension. From an abuse perspective, it raises the question of how to monitor for reward farming from these posts that are invisible from the website layer.

No answers right now, but these are questions that need thought.

Polling time for database_api/list_comments

In experimenting with the list_comments call, I ran into an interesting scenario. On my computer, this shell script runs in about 5 seconds:

#!/bin/bash
endDate=$(date +"%Y-%m-%dT%H:%M:%S")
startDate=$(date -d "+7 days" +"%Y-%m-%dT%H:%M:%S")
while [[ ! "${startDate}" < "${endDate}" ]]
do
   streamedPosts=$(curl -s --data '{"jsonrpc":"2.0", "method":"database_api.list_comments", "params": {"start":["'$startDate'","",""], "limit":1000, "order":"by_cashout_time"}, "id":1}' https://api.steemit.com | jq -Src '.result')
   lastCreatedDate=$(date -d "$(echo ${streamedPosts} | jq -Src '.[][].created' | tail -1)")
   startDate=$(date -d "${lastCreatedDate} + 7 days" +"%Y-%m-%dT%H:%M:%S")
   while IFS=, read -r author permlink created depth; do
       # Process each line of streamedPosts here
       echo "Author: $author, Permlink: $permlink, Created: $created, Depth: $depth"
   done < <(echo "${streamedPosts}" | jq -Src '.[][] | [( .author, .permlink, .created, .depth )] | @csv' | sed 's/\"//g')
done

But this one takes about 2 minutes:

#!/bin/bash
endDate=$(date +"%Y-%m-%dT%H:%M:%S")
startDate=$(date -d "+7 days" +"%Y-%m-%dT%H:%M:%S")
while [[ ! "${startDate}" < "${endDate}" ]]
do
   streamedPosts=$(curl -s --data '{"jsonrpc":"2.0", "method":"database_api.list_comments", "params": {"start":["'$startDate'","",""], "limit":1000, "order":"by_cashout_time"}, "id":1}' https://api.steemit.com | jq -Src '.result')
   lastCreatedDate=$(date -d "$(echo ${streamedPosts} | jq -Src '.[][].created' | tail -1)")
   startDate=$(date -d "${lastCreatedDate} + 7 days" +"%Y-%m-%dT%H:%M:%S")
   while IFS=, read -r author permlink created depth; do
       # Process each line of streamedPosts here
       if [[ "$(date -d "${created}" +"%Y")" -gt 2016 ]]
       then
          echo "Author: $author, Permlink: $permlink, Created: $created, Depth: $depth"
       fi
   done < <(echo "${streamedPosts}" | jq -Src '.[][] | [( .author, .permlink, .created, .depth )] | @csv' | sed 's/\"//g')
done

The only difference is the line by line handling of the output in order to get rid of extraneous data when the output wraps around to the blockchain's first posts. From the output behavior, I suspect that this has to do with shell buffering, but I'm not sure. Thoughts: 1.) It's bizarre that the API call loops around to 2016, instead of just terminating at the end; and 2.) I haven't (yet) noticed any equivalent slow down in JavaScript, but it's something to be aware of.

Geopolitics and Tron AI

When the 2020 US election happened, it was in the middle of the COVID pandemic, and I completely failed to anticipate the impact that that year's election would have on the crypto sector. During the last four years, the US SEC basically declared war on the crypto sector. As a result of the crackdown, I think that worldwide crypto innovation stalled in many ways.

All signs are that the new administration is going to be more welcoming to crypto-based innovation, so I am guardedly optimistic about the sector in general during the next four years.

It would be nice to know if the change in geopolitics will have any impact on Steem development plans from Steemit and/or the @tronfoundation. We have some hints from the Twitter account for @justinsunsteemit. Specifically, he mentioned investments of $75 million in the Trump-associated World Liberty Financial and he also announced his expectation for "groundbreaking" AI innovation between Tron and Steemit. (I also posted about this here and here.) I asked Grok what that innovation might be, and here's the reply:

Based on Justin Sun's post on X (formerly Twitter) where he mentions groundbreaking AI development on Tron and Steemit, here are some potential plans he might have in mind:

  1. Integration of AI for Enhanced User Experience: Given the trend towards personal AI and the integration of AI in blockchain platforms, Justin Sun could be planning to enhance user interaction on Tron and Steemit by embedding AI technologies that personalize content, improve user engagement, or optimize platform operations. This could involve AI-driven content curation, user behavior prediction, or even AI-powered moderation tools to foster a more engaging and secure community environment.

  2. AI-Powered Smart Contracts: Tron has been known for its focus on smart contracts, which could be enhanced by AI to create more dynamic, adaptive, and efficient contracts. This could include AI algorithms that automatically adjust contract terms based on real-time data or user behavior, potentially revolutionizing decentralized finance (DeFi) applications on Tron.

  3. Decentralized AI Agents: There's a growing interest in decentralized AI, where AI agents operate autonomously within blockchain ecosystems. Justin Sun might be planning to introduce AI agents on Tron and Steemit that could interact with users, manage content, or even participate in the ecosystem's governance, aligning with the trend of AI agents in crypto.

  4. AI for Content Creation and Curation: On Steemit, AI could be used to assist in content creation, providing tools for users to generate or enhance their posts, or to curate content more effectively, making the platform more attractive for creators and consumers alike. This could be an extension of what's already being discussed in the blockchain community regarding AI's role in content platforms.

  5. Partnerships and Ecosystem Growth: Considering Tron's strategy of leveraging partnerships, Justin Sun might be looking to collaborate with AI technology companies or developers to bring cutting-edge AI solutions to the Tron and Steemit ecosystems. This could involve partnerships similar to those with Samsung or BitTorrent, but focused on AI innovation.

  6. AI in Blockchain Security and Compliance: With AI's capability in data analysis, security, and compliance, there could be plans to use AI for enhancing the security of transactions on Tron, or for compliance checks on Steemit, ensuring that the platforms remain competitive and trustworthy in the blockchain space.

Given Justin Sun's background and the trends in blockchain and AI integration, these initiatives could significantly advance Tron and Steemit's position in the market, making them leaders in the fusion of AI and blockchain technologies.


In his inauguration speech, President Trump referred to January 20, 2025 as "Liberation Day". Without commenting on other aspects of American politics, I do hope that this will turn out to be true for the global cryptocurrency economy.

Conclusion

This post covered my recent programming activities from December 29 until today and it also described some of my plans for the future.

The main area of focus has been tag-following in the Steem Conversation Accelerator (SCA). Moving forward, before adding new features, I'm planning to rearchitect the tool for faster performance and easier usage.

Finally, the post contains reflections on some aspects of the Steem ecosystem as well as on the current moment in history. With the SEC leadership transition from Gary Gensler to Paul S. Atkins (presumptive nominee), perhaps we can hope to see a global increase in crypto-related innovation.


Thank you for your time and attention.

As a general rule, I up-vote comments that demonstrate "proof of reading".




Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.


image.png

Pixabay license, source

Reminder


Visit the /promoted page and #burnsteem25 to support the inflation-fighters who are helping to enable decentralized regulation of Steem token supply growth.

Coin Marketplace

STEEM 0.22
TRX 0.24
JST 0.037
BTC 102298.05
ETH 3247.59
SBD 4.70