AWS DeepRacer Community Log Analysis Challenge
In the Community it is important to us to help new members when they struggle, enable them when they want to contribute and support them in progressing from passive to active contributors. This is why we have started community challenges in which we can have fun, learn and help each other.
Source: Pixabay.com
In September the challenge was about racing - go as slow as you can. One thing we've learned is that there is an upper limit on the amount of steps the car can take while racing. While not a very useful information, it is one of the things we notice as we go more in depth with racing.
In October, however, it will not be (mainly) about racing. As you may have noticed, I have a thing for log analysis. It's not like I'm an expert with it or anything, it just clicked. The moment I realised what Jupyter Notebook allows me to do and what the log-analysis folder in AWS DeepRacer Workshop repo does, I kept running it, adding to it, modifying things as much as I could and worked hard to give it back to the community. Also occasionally I would get suggestions about what to add or what to fix. The main complaint was about readability. I agree, that's why yesterday I released an big update to descriptions and reorganised the layout of it.
It's mind blowing to see the AWS DeepRacer Community grow. We've had people join from all around the world, with varying experience in technologies needed to traing or to run the training locally. They started with the most basic questions and then the difficulty level only kept growing. I think the most rewarding moment to me as a member was when I saw those members who struggled the most initially help newer newstarters. An excellent pass-it-on game.
Now it's time to give you a chance to contribute to the projects that we have. The tool of choice for this month's challenge is log analysis tool.
Consolidation
Before I present the challenge, I would like to mention that this challenge also marks a start of consolidation of community projects under one group on GitHub.
We acknowledge that fragmentation can cause a lot of confusion, therefore we would like to move all the main projects under the community group.
The new Community fork of the AWS DeepRacer Workshop repository isn't currently used by deepracer-for-dummies, it's still my Breadcentric fork in there, but it's on purpose. I want to ensure you have a stable base to work on throughout the challenge. After that I will ask Alex to point his repository at the new place.
As the time progresses, we will gradually migrate all of the Chris' deepracer repositories as well. The guys at ARCC are also open to this idea. We will work to provide a consolidated version of the local training to simplify the setup part. It won't happen overnight, obviously, but we believe such a long term strategy will simplify onboarding new community members, introducing them to our existing projects, and creating new ones so that we can use AWS DeepRacer for what I believe AWS created it in the first place: spark curiosity in many heads, give an impulse to learn new things and bring people together (OK, I'm not sure if they intended that, but here we all are - it has happened).
The challenge
The goal is simple: prepare an improvement to the log analysis tool and submit it as a pull request on GitHub. You have two weeks to do it. Then it's up to the community to select the winner through votes on the #league-contest channel on the Community Slack. The winner will receive a $100 AWS credit.
Taking part
To take part make sure to go to https://contest.deepracing.io/ , read the rules, register, prepare the change and submit it as a pull request to the enhance-log-analysis branch of the code before the voting phase.
Remember that the pull request needs to contain your race name and the following text:
This change has been submitted as part of the AWS DeepRacer Community Log Analysis Challenge. To join the community go to http://join.deepracing.io
You also need to complete a lap in the race before the end of the month - we rely on AWS verification of users and we assume that if you have a lap in the race, you can take part in the challenge as far as eligibility criteria are concerned. It doesn't have to be a fast lap, but your username has to be in the leaderboard.
Votes
All community members vote. Before the voting phase starts, we will publish all submissions in the Community Slack and you will be able to vote there by commenting in a given submissions thread and clearly stating that this is your vote. Remember that you can vote for one submission only - if you vote more than once, your votes will not be counted.
There are also special votes: I get extra votes (five of them) that I will report to Lyndon at least 24 hours before the end of the voting phase. They will not be made public to not influence your votest and they will be given before you finish voting so that I cannot use them to block your selection of the winner in the last minute.
Rewards
We will deliver it by email after the winner accepts it. If they don't, we'll give it to the next person down the list.
- 1st place: $200 AWS credit
- places 2-3: $100 AWS credit
- places 4-10: $50 AWS credit
This has changed since the initial announcement as we managed to get wonderful support from AWS. Thank you, you rock!
Dates
The submissions will be accepted between 00:00 BST on Monday 7th of October 2019 and 20:00 BST on Wednesday 23th of October 2019.
The voting will take place between 00:00 BST on Thursday 24th of October 2019 and 23:59 GMT on Thursday 31st of October 2019.
We are working hard to get the registration up for Monday but you can start working on your submission right now, so this should not be an issue.
Help
I understand that it may be difficult to get started, that's why I encourage you to ask for help in the community.
Below you will find some questions with answers. They include a couple ideas of what you could try and improve or add. There also is an example showing how to create a pull request for submission. If however your question remains unanswered, feel free to reach out to me.
If you want to consult your idea that you would like to use, also ask, I will do my best to help.
Make sure your proposal can be executed. If you struggle to get it into a releasable state but it still kind of works, you can still submit it. If you struggle to make it reliable enough, it may be beneficial if you present the outcome you wanted to achieve.
Q&A
What is the purpose of this challenge?
- Promoting contributions to projects one benefits from
- Encouraging AWS DeepRacer Community members to contribute to the community’s projects
- Encouraging learning of tools used for data analysis
Why is the challenge about log analysis?
I’ve spent a lot of time using the notebook provided by AWS and learned a lot from it, but this is not a complete solution, and it never will be in my opinion. Each user has their own expectations and perspective. Jupyter notebook is a great way to enable expansion of the document. I have learned lots great tools this way, and I have learned to race better. Here’s your chance to do the same and get rewarded for it.
Why is the repository address different from the one provided with deepracer-for-dummies?
We, the AWS DeepRacer Community Pit Crew, have reached a conclusion that the community would benefit from a central space to host the repositories used and developed within community. This will let us benefit from having multiple maintainers of central repositories.
https://github.com/aws-deepracer-community/aws-deepracer-workshops is the first one to be moved to that space.
Additionally it will guarantee that all contestants have the same base version on top of which they can work and I will be able to work on my changes without interfering the challenge. Once the challenge is over, I will ask Alex and ARCC guys to point their solutions at the new repository.
Why do you want a Pull Request?
Our intention is to integrate submitted solutions into the Log Analysis utility for all to benefit from.
What is a Pull Request?
A Pull Request is a way of informing the maintainer of a project that you have prepared a change for all to benefit from.
I don’t know how to make a fork/Pull Request
Scroll down for a submission example. You can also read on one of my previous posts where I presented an example of contributing to a project through GitHub.
I don’t know how to work with log analysis utility
Have a read: https://codelikeamother.uk/analyzing-the-aws-deepracer-logs-my-way and feel free to ask in the #training-log-analysis channel
I don’t know how to contribute
Great, you have two weeks and a bit to learn.
I cannot get my idea to work
This sometimes happens. Make sure you provide as detailed description as possible. You don’t have to integrate the suggestion with existing notebooks, you can add a new one. We will work on it later together to introduce the changes to the repository in the most convenient way from the users’ perspective.
Why are there special votes? Why are there all those strange rules associated with them?
I decided to introduce this because as a reward sponsor I would like an extra say as to which ideas I like the most. We decided to keep them hidden so that I don’t suggest to others which submissions I like the most and decided to send them to Lyndon well before the end of votes so that I cannot cause a last minute change of leaders.
I don't know what I can do
There are so many options:
- add, modify or replace graphs
- add a functionality not present yet - maybe something around the action breakdown?
- finish an incomplete feature (like parameters in new_reward function)
- refactor an existing piece of code
- fix a bug
- add/improve descriptions
If you have a trusted community member, you can discuss your idea with them.
Sample pull request
While I have created an example of contributing to a project through GitHub, I would like to lead you through the PR submission for our challenge.
At this point I am assuming you have some knowledge about git and have registered with https://github.com - I am using my other account in this example.
I also encourage that you set up ssh keys - I know it may sound scary, but it's reasonably well documented and will simplify your interaction - if you use http urls when working with git, you need to provide a username and password regularly. Interacting over ssh means you use your ssh keys to authenticate and you don't need to provide the password all the time.
What we will have to do:
- create a fork of the repository
- clone the fork into your computer
- prepare a branch off the
enhance-log-analysisbranch - apply some change
- push it to your fork
- raise a pull request
1. Create a for of the repository
When you're logged in on github, you can go to a project page and fork it. A fork means you create a copy of the repository in your own workspace. You can apply your own changes in there and either offer them to the main repository or carry on working with your own line of code.
When you go to https://github.com/aws-deepracer-community/aws-deepracer-workshops creating a fork is as simple as this:
A couple seconds later you've got your fork ready. What's nice, it also copies branches:
The big green button tells you what you can do to clone the repository to work with locally:
As I've mentioned, without the certificates it provides http URL for cloning. I will use ssh in the next steps.
2. Clone repository
I run this code:
git clone [email protected]:tptak/aws-deepracer-workshops.git
and received following output:
~/challenge$ git clone [email protected]:tptak/aws-deepracer-workshops.git
Cloning into 'aws-deepracer-workshops'...
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 2248 (delta 7), reused 17 (delta 7), pack-reused 2230
Receiving objects: 100% (2248/2248), 38.69 MiB | 13.52 MiB/s, done.
Resolving deltas: 100% (738/738), done.
Now let's see what we've got in here:
~/challenge$ cd aws-deepracer-workshops/
~/challenge/aws-deepracer-workshops$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
~/challenge/aws-deepracer-workshops$ git branch -v
* master 2ed5ce7 Add routes for NYC, China, and Mexico
~/challenge/aws-deepracer-workshops$ ls log-analysis/
cw_utils.py 'DeepRacer Log Analysis.ipynb' intermediate_checkpoint log_analysis.py logs simulation_episode tracks
As you can see, the new branch has not been pulled automatically, but we'll change it soon.
3. Prepare a branch off the enhance-log-analysis branch
Let's go onto the right branch:
~/challenge/aws-deepracer-workshops$ git checkout enhance-log-analysis
Switched to branch 'enhance-log-analysis'
Your branch is up-to-date with 'origin/enhance-log-analysis'.
~/challenge/aws-deepracer-workshops$ ls log-analysis/
cw_utils.py Evaluation_analysis.ipynb log_analysis.py README.md simulation_episode track_utils.py
'DeepRacer Log Analysis.ipynb' intermediate_checkpoint logs reward tracks Training_analysis.ipynb
~/challenge/aws-deepracer-workshops$ git branch -v
* enhance-log-analysis 757fa87 Improve descriptions for Evaluation_analysis.ipynb, reorganise slightly and add small code improvements
master 2ed5ce7 Add routes for NYC, China, and Mexico
Not bad, eh? Now let's create a branch to work on:
$ git checkout -b sample-contribution
Switched to a new branch 'sample-contribution'
4. Apply some change
I would like to update a Training notebook with some added description. I also think we should have a table with slowest complete episodes in it.
I already have a python venv that I work with so I will just activate it and start jupyter-notebook:
:~/challenge/aws-deepracer-workshops$ source ~/venv/bin/activate
(venv) ~/challenge/aws-deepracer-workshops$ cd log-analysis/
(venv) ~/challenge/aws-deepracer-workshops/log-analysis$ jupyter-notebook
[I 22:08:08.947 NotebookApp] The port 8888 is already in use, trying another port.
[I 22:08:08.947 NotebookApp] The port 8889 is already in use, trying another port.
[I 22:08:08.951 NotebookApp] Serving notebooks from local directory: /home/user/challenge/aws-deepracer-workshops/log-analysis
[I 22:08:08.951 NotebookApp] The Jupyter Notebook is running at:
[I 22:08:08.951 NotebookApp] http://localhost:8890/?token=db01f3022d751f321bdffd5e305f0313e3ce22416e36eb6d
[I 22:08:08.951 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 22:08:08.956 NotebookApp]
To access the notebook, open this file in a browser:
file:///home/user/.local/share/jupyter/runtime/nbserver-7111-open.html
Or copy and paste one of these URLs:
http://localhost:8890/?token=db01f3022d751f321bdffd5e305f0313e3ce22416e36eb6d
It also opens the notebook for me in my browser.
I open the Training_analysis.ipynb and add the desired change:
Now we can save and shut down Jupyter notebook (with pressing Ctrl+c twice in the terminal). Let's see the changes:
(venv) ~/challenge/aws-deepracer-workshops/log-analysis$ git status
On branch sample-contribution
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: Training_analysis.ipynb
Untracked files:
(use "git add <file>..." to include in what will be committed)
__pycache__/
no changes added to commit (use "git add" and/or "git commit -a")
We need to add the notebook to a change set and to commit the change set:
(venv) ~/challenge/aws-deepracer-workshops/log-analysis$ git add Training_analysis.ipynb
(venv) ~/challenge/aws-deepracer-workshops/log-analysis$ git commit -m "Added a list of slowest laps"
[sample-contribution 8c63e13] Added a list of slowest laps
1 file changed, 251 insertions(+), 85 deletions(-)
Great! We have the changes saved locally, now we need to push them to our fork.
5. Push the change to your fork
The reason why changes aren't yet available on GitHub is the distributed nature of git - your clone of a repository, sitting on your computer, is just like a fork of the upstream repository, sitting in your working space. You can apply changes to it and never send them. When you want to, you need to explicitly push out the change. Let's do it:
(venv) ~/challenge/aws-deepracer-workshops/log-analysis$ git push
fatal: The current branch sample-contribution has no upstream branch.
To push the current branch and set the remote as upstream, use
git push --set-upstream origin sample-contribution
Busted. The repository doesn't know where it should push the change - we have created the branch only locally. I'm going to follow advice shown in the response:
(venv) ~/challenge/aws-deepracer-workshops/log-analysis$ git push --set-upstream origin sample-contribution
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.36 KiB | 63.00 KiB/s, done.
Total 4 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
remote:
remote: Create a pull request for 'sample-contribution' on GitHub by visiting:
remote: https://github.com/tptak/aws-deepracer-workshops/pull/new/sample-contribution
remote:
To github.com:tptak/aws-deepracer-workshops.git
* [new branch] sample-contribution -> sample-contribution
Branch 'sample-contribution' set up to track remote branch 'sample-contribution' from 'origin'.
Now you can see our branch on the list:
You can also see our commit:
6. Raise a pull request
Last step. One tricky moment here is that by default pull requests are pointed at the master branch, not the one we have branched off. Just remember to change it.
Read the race rules - there is a statement that you need to put in your Pull Request that declares your participation in the challenge. Also provide your racing name. Something like the following:
My racing name is BananaSoyDecafLatteWithJustATeenyWeenyChocolatePowderOnTop
This change has been submitted as part of the AWS DeepRacer Community Log Analysis Challenge. To join the community go to http://join.deepracing.io
On the branch view do the following:
And that's it, pull request has been raised: https://github.com/aws-deepracer-community/aws-deepracer-workshops/pull/1
You can go back at a later point and edit your pull request if needed.
It wasn't that bad, was it?
Summary
As always, I encourage you to join the AWS DeepRacer Community racing and learning. Not much time left to race this season, but hopefully you'll be able to use in this challenge what you've learned so far, so head to https://contest.deepracing.io/ and register!
Good luck and have fun!