I agree on all of your points. Personally, I'm much more of a lurker than a commenter, so questions in an article wouldn't do a great job at measuring my attention. The concept definitely needs to be improved & extended. Of course, there's a whole area of research into attention tracking, so I'm sure there are tons of possibilities for web site owners that I'm not even aware of. (things come to mind like page view times, mouse movements, click tracking, etc...)

As to the type of questions, the first thought is that they should be related to the content, but there are other possibilities. For example, there could be generic questions about finding related information elsewhere on the Internet or even in other Steem posts. i.e. "Find a Steem post that relates to this one and post a reply to describe some things that the two posts have in common". I thought of some others earlier, but now I'm drawing a blank.

You're also right that people would use AI to find the questions, and try to game the system. That's another reason why I thought of embedding the questions as images. It's not impossible to read text from images, but that would make it harder.