Dealing with regular expressions and investigating memo encryption for my offchain-content tool

As I've chronicled in a few posts, I've been working on a Node.js tool for easily posting stub posts pointing to offchain content onto the blockchain. Unfortunately it has been slower going than I had hoped.

Regular Expressions

Back at my first real job the team I was on used Perl for most of our custom scripts and tools, and Perl has pretty solid support for regular expressions, so I got used to using that functionality extensively. That experience locked in my expectations, so when I use any other language I often get frustrated when using regular expressions because it almost always turns out to be more awkward or difficult than I expect. When my offchain-content tool starts up it needs to scan the blockchain for its previous posts so that it can 1) avoid posting duplicates, and 2) delay posting anything new if the previous post is less than five minutes old. In order to parse the information out of the previous posts on the chain my first instinct was to use regular expressions. Then I had second thoughts, knowing that I've been frustrated every time I've tried to use Javascript's regular expression functions in the past. But then I figured that the more virtuous course of action would be to bite the bullet at figure out how to do it with regular expressions, since that is the right tool for the job. And I'm glad I did, because I figured out that Javascript has exactly the functionality I wanted and I had just never seen it before.

Named Groups

I knew that Javascript could detect regular expression matches, but I didn't think that it could extract parts of the text that match. So, for example, I didn't want to just be able to see that something looked like the markup text for displaying a link, I wanted to be able to pull the URL out of it at the same time. I thought that I would have to run the regular expressions and use string manipulation functions to pull out the relevant substrings, but it turns out that you can extract the substrings that match things inside parenthesis groups in your regexp, plus you can even attach names to each group and extract the set of matches as an object with keys based on the names.

So now my code for parsing previous posts has a regexp like this:

const regexpPostStructure = />\s*#\s*\[(?<title>[^\]]+)\]\s*\((?<url>[^\)]+)\)\s*>\s*\!\[preview\s+image\]\((?<imageurl>[^\)]+)\)\s*>\s*(?<previewtext>[^]*)\*\(This post is just a link to content not posted on the chain\. Author rewards are 100\% redirected to null\. h\/t\s+\@(?<referrer>\S+)\)\*/m;

And when I use the "match" function it returns an object with a groups property that is an object with the properties I named in the regexp:

    await client.database.getDiscussions('blog', {tag: startAuthor,
                                                  limit:paginationSize,
                                                  start_author: startAuthor,
                                                  start_permlink: startPermlink
                                                 })
    .then((response) => {
      response.forEach((post) => {
        // make sure it's not a reblog
        if (post.author == startAuthor) {
          startPermlink = post.permlink;
          matches = post.body.match(regexpPostStructure);
          if (matches) {
            var postObj = {
              steemiturl: post.url,
              title: matches.groups.title,
              canonicalurl: matches.groups.url,
              imageurl: matches.groups.imageurl,
              referrer: matches.groups.referrer,
              datetime: new Date(post.created)
            }
         }
       }
    });
  });

(Image from Pixabay)

Did you get my memo?

I also need to implement "login" style functionality for the tool, because I only want to post on behalf of people who have enough SP to be able to meaningfully upvote the post once it gets on the chain (and not be a tool to post spam on behalf of malicious users). I figured that this would be a pretty simple thing that lots of tools would need, but I haven't seen any straightforward documentation laying out the best practices. After trying to poke around in a few places, my understanding of the "keychain" approach is that it uses the cryptography features of the "memo" in a transfer transaction. When you transfer STEEM or SBDs to someone and start the memo on the transaction with a # symbol the rest of the memo is encrypted with the recipient's public key, so only someone who has access to the recipient's private key will be able to decrypt it. So the idea is that your site can use the functions that create encrypted memos to encrypt some info as a token (like a temporary password), send it to the recipient who claims to be user XYZ, and if they can send the unencrypted version of the token back to you that proves they are who they say they are. So now that I've understood that (and dipped my toes into understanding a little about how cryptography and keys work) I think that's a sound principle. But I still need to figure out how to actually do it in my code.

I saw on dsteem's issue page that it has an open issue asking about memo encoding. And it seems like the steemit wallet code uses steem-js. So I thought I might learn something by doing a few experiments. Unfortunately my experiments so far are not working at all. I haven't used steem-js before, so maybe I'm not setting it up properly. But it seems like dsteem is crashing when I try to do transfers, so that's frustrating, because I can't get to figuring out that part that I'm interested in until I debug why it's not working at all.

client.database.getDiscussions('blog', {tag: process.env.STEEM_ACCOUNT, limit:1})
.then((response) => {
  console.log("Finished reading", response);
  return client.broadcast.transfer({from: 'offchaincontent',
                                    to: 'danmaruschak',
                                    amount: '0.001 STEEM',
                                    memo: memo: 'testingABC'
                                  },
                                    process.env.STEEM_PRIVATE_KEY)
})
.then((result) => {
  console.log("dsteem transfer 1 done", result);
  return client.broadcast.transfer({from: 'offchaincontent',
                                    to: 'danmaruschak',
                                    amount: '0.001 STEEM',
                                    memo: '# testingDEF'
                                  },
                                    process.env.STEEM_PRIVATE_KEY)
}).then((result) => {
  console.log("dsteem transfer 2 done", result);
});

[the path to where I'm doing this experiment]\node_modules\dsteem\lib\crypto.js:315
        const signature = key.sign(digest);
                              ^

TypeError: key.sign is not a function

So I guess I will try to keep debugging to figure out what's going on here (or if anybody else has any insight feel free to comment). I'm getting somewhat frustrated that the libraries and documentation don't seem to be better maintained (I should probably make sure that Javascript library support and maintenance is mentioned in at least one DIP proposal so there's a chance someone will take that over).