STEEM API rate-limiting is breaking stats script: Daily flag-war stats are down; possibly permanently.

If you are using the STEEM-API, you may have noticed some nodes are giving an unusual amount of 4xx errors at the moment. The daily flag-wars stats script had not been finishing its daily task a number of days in a row due to such errors from top API nodes, and as these things normally resolve themselves, I thought I could just wait it out. That was, untill I noticed this messages on Discord:

and a little later, this discussion:

Turns out, the problems my script, and that of others, are experiencing right now, isn't just some temporary problem; it is actually by design. And as scripts like mine that try to scrape together meaningful statistics from API nodes do a hell of a lot API calls by design themselves, they are basically killing of this type of scripts by implementing rate limiters like this. Let me describe what my script does that is incompatible with the design of the rate limiter currently being used by some of the top nodes:

• It does a quick binary search to find the first block of the last day for what voting has closed.
• It requests that block plus all consecutive blocks that were created on that day.
• For each post created in the block it requests the list of votes made on that post.
• For each account involved in flagging on the day in question, it requests user info
• It generates a report like this one, with a more complete graphical representation like this one

However, relaxed @justinw may think the rate-limiting is, requesting the votes made for every post made 8 days ago is an absolutely required part of the stats script, and a part that doesn't currently make it to one hour of posts, let alone a full day before hitting the relaxed rate limit.

Seems that at this moment it would still be possible for my script to simply blacklist the API-nodes that implement rate limiting, but it is likely the rate-limiting code will soon propagate to other nodes as well. If it does, that basically means the rate-limit feature will be a dead blow to my stats script and likely those of others who don't have the resources to run their own full node or a database server.

So for now, I'm sitting this one out. Hoping Steemit would reconsider this feature or that witnesses running their own API nodes will consciously decide not to propagate this script-killing feature to their nodes.

Until then, the daily flag-war stats will be down. Possibly permanently if Steemit keeps this feature and this feature were to propagate to other API nodes currently not (yet) affected.