Student: Hussain Khan (@fear-the-reaper)
Mentor: Shubham Pandey and Eshaan Bansal
Organization: The Honeynet Project
Project: Intel Owl
Tag: Information Security
Intel Owl is an Open Source Intelligence or OSINT solution to get threat intelligence data about a specific file, an IP, or a domain from a single API at scale. It integrates a number of analyzers available online and is for everyone who needs a single point to query for info about a specific file or observable.
I propose making a robust Go client library for OSINT Threat Intelligence Platform IntelOwl that easily communicates with their API. The Intelowl Go SDK will allow developers to communicate with the API so that they can easily develop and integrate IntelOwl with their own automated scripts, tools, and services.
So the main objective was to develop a robust Go client library that is easy to use for developers and easily extensible for adding new features.
Pre GSoC Commits
List of pull requests merged before GSoC’s coding period:
GSoC Tasks and Deliverables
I made 20 commits and over 20 pull requests in go-intelowl. The following major tasks were completed and maintained over time. You can see the release here.
1. Proper project structure and automated workflows
Previously the SDK was just a client.go and a go.mod. This lacked a proper structure and placement where adding new features or revisions would be a hassle. Now the project has a proper structure where you can easily add new features, examples, tests, and workflows. Furthermore I added Issue and PR templates that closely resembles IntelOwl’s. In addition to this, I implemented a proper CI/CD so that with every pull request and push a linter, unit tests, dependabot, and codeql would run.
2. Breaking down the API endpoints into service objects
Previously the SDK was only a client struct with every endpoint implemented as a method. This gave rise to repetitive code in most methods and couldn’t be easily extensible or maintainable as you’d need to keep track of many methods so debugging would be hard if not impossible.
Working and solution
For the above problems we first divided the API into 7 service objects where each has its own optional parameters. Furthermore, to adopt the builder pattern we opted on making an internal method for the client to easily facilitate the service objects. Lastly, we provided methods to easily instantiate the client through a JSON file representing the client’s own optional struct. In addition, to that, we built our own logger using logrus. We opted to use logrus as it gave us great flexibility.
3. Proper and thorough unit testing of each service object
Now in order to fully check and test the service objects we needed to perform adequate unit testing that can be easily added and flexible.
I and Shubham discussed this at length on making a robust testing suite. Towards this, we looked at various methods of how tests are made in Go. We encountered a variety of problems as #37
For this to be implemented we had to change our perspective and approach on how to do things
Working and solution
To achieve the aforementioned objectives we used Go’s httptest to properly mock IntelOwl server. Furthermore, we adapted Table Driven tests so that we can run multiple tests for an endpoint under a single method.
I’m grateful to our mentor, Shubham, who spent a great amount of time and effort to solve most of the above-mentioned issues which significantly sped up the process.
4. Writing godoc documentation and proper examples
The last part was writing documentation, and easy-to-read examples to help developers understand to easily use our humble SDK. Now Go provides a way to document your Go package using the godoc tool where it parses Go comments and presents them on a web page.
I’m very glad to have worked on this amazing project and would continue to do so in the future. It has been a major source of learning for me. Some of the things I hope to tackle in the future are
I would like to thank The Honeynet Project and Google Summer of Code for providing me with this opportunity. Special thanks to my mentors Eshaan Bansal and Shubham Pandey for being kind and helpful to me throughout this amazing journey.