We Created an Awesome Search Tool for Github Repos in Python

Background

Recently, we got a notification from Amazon that one of our secret keys, which was accidentally left in a public Github repo, was being abused. Since it wasn’t our own team doing the abusing, we decided to put a stop to this happening in the future. So, the idea here was to create a mechanism to search Github for any string variable that matched any of our secret keys. Naturally, you can use this to search any string variable.
We looked for some of the other apps out there that can already do this and we found gitrob, a Ruby-based tool which didn’t offer many integrations, and was a bit heavy on the code. It also doesn’t include much documentation or have any sort of community to turn to for help.
So, we decided to build our own tool – and it’s called Surch!

Changing the face of DevOps – automate all things with Cloudify.  Go

What is Surch?

Surch is an open source Python package that can be installed with command pip install Surch. Simply run your Surch search in the terminal. If a commit is found with a string matching one of your predetermined strings, you will see the log showing it on the screen and can proceed to manually delete the problematic commit, then notify Github to remove it using ‘sha commit’. NOTE: It’s up to you if you would prefer to change the credentials, but we would highly suggest it.

Building Surch

When we started talking about how to build this tool, our original thought process went something like this:

• We want one script that simply runs the search on Github organizations searching all repos within
• It needs to have the ability to run on public repos (private repos have passwords and are more complicated to handle)

For our purposes, we added a Jenkins job to automatically run Surch once per day.
By the time our first release came around, it was actually a lot more sophisticated, with the following additional features:

• You can choose to only search a specific repo by adding it to the search string
• You can choose to only search a specific user’s repo by adding it to the search string
• Options to include or exclude repos based on your search parameters
• Integrations with PagerDuty for instant notifications
• Vault integration – Vault allows for safe secret or password storage, and we integrated it to automatically update Surch when a secret key is updated

Future Features

• Integration with Slack
• Integration with AWS Lambda – git would notify Lambda that a commit was made and Surch automatically conducts a search on that specific commit only
• The ability to run searches on private repos
• Automate the handling and fixing of problematic commits

Surch Basics

1. Install with pip install Surch
2. Run Surch repo REPO_URL -FLAGS to search a specific repo
3. Run Surch org ORG_NAME -FLAGS to search all repos in that organization
4. add --incude-repo REPO_NAME to search only that specific repo in an organization and add the same flag multiple times to search many repos

For more info on this project, see the docs. All contributions are appreciated!