Create script to generate Flatpak manifest from PIP packages

in #utopian-io7 years ago (edited)

The tool to generate flatpaks, flatpak-builder is designed to be sandboxed and generate reproducible results; The downside to this is that some existing tooling such as npm or pip don't necessarily map well to this as they by default download dependencies and packages generally have a lot of dependencies which you wouldn't want to list by hand.

So the solution to this is simply to have automated tooling to generate a json file containing all of the dependencies and include them in your manifest.

The information we need to do this is the list of packages, their sha256, and a URL to download them. Thankfully pip has a download option which resolves the list of all dependencies and downloads them for us so we can get the hash ourselves.

To get the download URL we have to use pypi's API at https://pypi.python.org/pypi/{package_name}/json and find the releases url that matches the filename pip grabbed earlier which is easy enough.

Now that flatpak-builder can reliably download static versions of dependencies we need to install it. The pip install tool does allow looking up packages in a local directory so we can pass it --no-index --find-links "file://${PWD}".

This solution did not turn out perfectly though. For a reason I've yet to track down some packages, for example mypy, have actual dependencies that are not listed or pulled in which is frustrating and currently just worked around by allowing you to pass multiple packages at once so you can manually specify anything missing which seems acceptable for now.

The end result looks like this:

{
    "name": "jedi",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links \"file://${PWD}\" --prefix=/app jedi"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://pypi.python.org/packages/14/3b/15cfd9c7a8bd9e3a2fe956e20fbc4e7c5768e06aea347d8eb68a05a71653/parso-0.1.1.tar.gz",
            "sha256": "5815f3fe254e5665f3c5d6f54f086c2502035cb631a91341591b5a564203cffb"
        },
        {
            "type": "file",
            "url": "https://pypi.python.org/packages/d2/41/430b325e411d564b1afc37bc7286c3549f4c415ada750a617fb1943c593d/jedi-0.11.1.tar.gz",
            "sha256": "d6e799d04d1ade9459ed0f20de47c32f2285438956a677d083d3c98def59fa97"
        }
    ]
}

Merge request: https://github.com/flatpak/flatpak-builder-tools/pull/6



Posted on Utopian.io - Rewarding Open Source Contributors

Sort:  

Thank you for the contribution. It has been approved.

You can contact us on Discord.
[utopian-moderator]

Do you know why the payout of this post is so much lower than the others?

EDIT: Nevermind it was just delayed.

Hey @tingping I am @utopian-io. I have just upvoted you!

Achievements

  • You have less than 500 followers. Just gave you a gift to help you succeed!
  • Seems like you contribute quite often. AMAZING!

Community-Driven Witness!

I am the first and only Steem Community-Driven Witness. Participate on Discord. Lets GROW TOGETHER!

mooncryption-utopian-witness-gif

Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x

Congratulations @tingping! You received a personal award!

1 Year on Steemit

Click here to view your Board of Honor

Do not miss the last post from @steemitboard:

Meet the Steemians Contest - The results, the winners and the prizes

Support SteemitBoard's project! Vote for its witness and get one more award!

Congratulations @tingping! You received a personal award!

Happy Birthday! - You are on the Steem blockchain for 2 years!

You can view your badges on your Steem Board and compare to others on the Steem Ranking

Vote for @Steemitboard as a witness to get one more award and increased upvotes!

Coin Marketplace

STEEM 0.15
TRX 0.16
JST 0.028
BTC 68123.34
ETH 2416.53
USDT 1.00
SBD 2.35