You are viewing a single comment's thread from:

RE: Learn Python Series (#14) - Mini Project - Developing a Web Crawler Part 2

in #utopian-io7 years ago

Since we are thinking algorithms eh, I think one way we could make this more dynamic, and to solve the "related users" problem going with the "recent-replies" method, is to do option 3, then 4. That way, we get totally unrelated and widespread users.

First, we try to fetch the new posts, then for each of these users, we carry out method 4 to obtain users in their recent replies. This might actually be able to crawl the full blockchain, assuming that each user on the blockchain is linked to/has commented on at least one other user on the blockchain.

Quick one though (probably unrelated ): I was really interested in the posts fetching algorithm you mentioned...
So our API returns the first 20 or 50 posts as the case may be. Then, onClick of the "next" button, it fetches the next 20 or 50. What parameter do you specify that tells the API to fetch second set of 20 or 50? Because I dont think I find any such specification in the Steem API. Is there a workaround I'm not seeing?

Sort:  
Loading...

Coin Marketplace

STEEM 0.25
TRX 0.20
JST 0.037
BTC 96252.72
ETH 3559.97
USDT 1.00
SBD 3.75