Mocking AWS S3 Scanner with Golang

By Effi Bar-She’an

Image for post
Image for post

Analyzing files on an AWS S3 bucket is a common task with many examples for doing so available on the Internet, however, doing it in a way that enables unit tests, is somewhat of a mystery.

So here’s a complete example of a golang client that:

  1. Downloads each file from an S3 bucket to a local filesystem
  2. Does some work
  3. Deletes the temporary file from your filesystem

And, it is testable!

Let’s take a look at what the client looks like:

The constructor accepts an AWS region and a local path to use for storing the temporary downloaded S3 files. accepts the bucket to scan and triggers the callback for each downloaded file.

S3Scanner Implementation

We start by creating an S3 service in the constructor:

The trick here is to use the interface instead of which is returned by the function. This makes it testable!

Note, that when you initialize a new service client (svc above) without supplying any arguments, the AWS SDK attempts to find AWS credentials by using the default credential provider chain. In our case verifies that and have non-empty values:

Now, we call using the that was created in the constructor, iterate the response and pass the files on to our callback.

Here is AWS Go SDK example, with our trick of using the interface to make it testable :)

Let’s take a look at the download function which uses the that we created in the constructor:

The Unit Tests

As mentioned above, our code is testable. Let’s see how.

We start by mocking S3 :

As you can see our mock inherits from and returns an object with a which represents the file path on S3.

To mock the function of we implement that returns mock S3 file data:

Let’s put all together:

See a full example in: https://github.com/Tufin/blog/tree/master/s3-scanner

Reference: Go SDK S3 Docs, AWS Go SDK Examples on GitHub

From the Security Policy Company. This blog is dedicated to cloud-native topics such as Kubernetes, cloud security and micro-services.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store