For now, I am using account history for the profile and filtering the votes. But this is very inefficient as I need to iterate through a lot of non-vote entries in order to collect a sufficient number of votes.

Here's an example using beem -

from beem.account import Account
acc = Account('singhpratyush')
for entry in acc.history_reverse():
  if entry['type'] == 'vote' and entry['voter'] == acc.name:
    # Do something with entry

The entry would look something like this -

{'account': 'singhpratyush', 'timestamp': '2019-01-10T15:21:48', 'type': 'vote', '_id': 'c3ebc9b2f2ce5b561d1a6c2873372020ff74e90c', 'author': 'ahmadnayan', 'trx_in_block': 34, 'voter': 'singhpratyush', 'trx_id': '453bdef21336689414e142dbf033e08848679d91', 'block': 29336541, 'op_in_trx': 0, 'weight': 10000, 'permlink': 'n9b6astn4ljaqw00x9lfckegyks65v81o585hg0fqfu01h1a7xw0zvzfpw0ft3hc20190110213037', 'index': 3404, 'virtual_op': 0}