Continuous integration of Github repo with S3 bucket using TravisCI

Overview

Continuous integration might seem to be very complicated at first, especially when it comes to coders, but don’t intimidate we will try to understand the concept of CI as simple as we can and then will try to implement continuous deployment of a Git repo on AWS S3 bucket using TravisCI.

To understand CI, imagine a scenario where your team is working on a project, committing changes to version control, and every time they commit a change, you want the code to be built, tested and deployed on S3. One way can be, you build the code locally, test it and then upload all static files in S3 manually or using awscli to accomplish the deployment task. But then doing all these manually is a bit overhead right? You have to do all these steps, every time a new commit occurs.

And this is where continuous integration comes into the picture.

Its a process of building, testing and deploying the code automatically every time team member checks into version control

In this article, we will see how we can integrate TravisCI with our project in Github to AWS S3. We will use following components:

Version Control : Git

Project Type : Node JS

CI tool : TravisCI

Cloud Object Storage : AWS S3

There are many CI tools available, but we will be using TravisCI because its free for all open source projects hosted on the GitHub and for the first 100 builds otherwise. You can find the list of available CI tools here

Steps to follow

Setting up Git Repo

Setting up Travis account

AWS Setup

Setting up S3

Creating IAM user for TravisCI

Test the IAM user has access to S3 bucket or not. (Optional)

Installing and using AWS CLI

Uploading files to S3 using aws-cli through IAM user

4. Configuring .travis.yml file in repo

Setting up Git Repo

Create a git repository and push your node js project code to it.

Setting up Travis account

Go to this link https://travis-ci.com/ and signup with GitHub account. Provide Github username and password -> Authorize to Travis -> Press on Activate button -> Choose either all repos or selected a repo -> Press Approve and Install.

Once you are in the home page, on your desired repository, press on setting button -> Enable Build only if .travis.yml file present.

AWS Setup

Setting up S3

Login to AWS console and create an S3 bucket.

Once the bucket is created, go to Properties tab and enable Static website hosting

Creating IAM user for TravisCI

Go to AWS console and search for IAM -> add a new IAM user

Give Programmatic Access

Copy Secret Access Key and Access Key ID, we are going to need it later.

Once the user is created, click on them and on the Permission tab, hit Add Inline permission link.

Open the JSON tab and past the following JSON. Make sure you replace the resources by your bucket name.

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "s3:ListAllMyBuckets",

            "Resource": "arn:aws:s3:::*"

        },

        {

            "Effect": "Allow",

            "Action": [

                "s3:ListBucket",

                "s3:GetBucketLocation"

            ],

            "Resource": [

                "arn:aws:s3:::your_bucket_name"

            ]

        },

        {

            "Effect": "Allow",

            "Action": [

                "s3:PutObject",

                "s3:GetObject",

                "s3:GetObjectAcl",

                "s3:DeleteObject",

                "s3:PutObjectAcl"

            ],

            "Resource": [

                "arn:aws:s3:::your_bucket_name/*"

            ]

        }

    ]

}

Click on create policy button, give a proper name and save.

Test the IAM user has access to S3 bucket or not

This step is optional, you may skip this step.

Installing and using AWS CLI

Let’s use AWS cli to login with our new user and uploading files to S3… This is for Ubuntu.

Now that you have installed python, use the command to install awscli sudo apt install awscli

Verify aws is installed, aws --version

Configure aws with a newly created user.

Type command aws configure

Provide Access Key ID, Secret Access Key that you have copied earlier.

Provide Default region name as us-east-1 (depending upon your configuration) and Default output as JSON.

Note: If some error occurs while installing awscli, please install python first.

Uploading files to S3 using aws-cli through IAM user

Create index.html file in a folder and cd to into it.

Use command aws s3 cp index.html s3://[bucket_name]/index.html to copy the file to s3 bucket. Don’t forget to replace the bucket name in the command.

Configuring .travis.yml file in repo

In TravisCI, builds are configured using .travis.yml file which contains the build tasks that will be executed on running the build.

Clone the existing repo and create .travis.yml file in root folder. Add following code into it.

language: node_js

node_js:

 - node

cache:

 directories:

   - node_modules

deploy:

 provider: s3

 access_key_id: "YOUR_ACCESS_KEY"

 secret_access_key: "YOUR_SECRET_KEY"

 bucket: "YOUR_S3_BUCKET"

 acl: public_read

 skip_cleanup: true

 local_dir: build_webpack

branches:

 only:

   - master

addons:

   apt:

     update: true

This config file is telling Travis to build the project as node project. Hence the language is specified as node_js

Replace “YOUR_ACCESS_KEY” and “YOUR_SECRET_KEY” with keys that you have copied earlier.

Change the bucket field as well.

local_dir field tells Travis to deploy only the specified directory (here build_webpack) to the bucket so that you won’t have to deploy your whole code.

apt -> update true : property tells to run apt-get update.

acl property can set Travis correct file permission. In this case it’s public. To read more here.

Now you can do changes in any file, say index.html. Build your project using command, say npm run build then commit the changes. Travis will detect the new commit and trigger the build for you.

You might have noted that we are providing Access Key and Secret Key in .travis.yml and then checking this file along with other changes. Doing this will reveal your key so it’s better to encrypt these keys first and then add them to .travis.yml file. You can accomplish this task in either of these two ways.

Download travis client -> Open terminal -> type command travis encrypt --add deploy.secret_access_key secret_access_key="YOUR_SECRET_KEY"

OR

(Easiest way) Open TravisCI web app -> login -> click on setting button near your repo -> look for environment variables section -> Add YOUR_SECRET_KEY, YOUR_ACCESS_KEY and YOUR_S3_BUCKET as variables and specify their proper values as well. Now, you can refer to environment variables from .travis.yml like this :

....

 access_key_id: "$YOUR_ACCESS_KEY"

 secret_access_key: "$YOUR_SECRET_KEY"

 bucket: "$YOUR_S3_BUCKET"

....

Hope this article will help you in CI next time. For more, please refer to TravisCI doc