Sending Nginx Access Logs to a Golang Server over syslog

By Effi Bar-She’an and some other guy

Steps:

  1. Writing a Golang UDP server
  2. Testing our UDP server using netcat (nc), a utility for reading from and writing to network connections using UDP (or TCP)
  3. Configure and run nginx to send access-logs to our Golang server
  4. Create a custom access-log format
  5. Parsing access-log json sent from nginx

Writing a Go UDP server

Testing our UDP server

In order to test our UDP server on mac we can use netcat (more info about it you can find in here):

Now, we are connected to our UDP server running on localhost on port 6060, so writing on the terminal will send packets to the server and vice versa. Mean, writing, for example “hola” on the terminal should result with a log in our Go server.

Tip, you can also use netstat to check if our server listen on udp:

Configure and run nginx

Let’s start with a simple ngnix config that listens on port 8085, functions as a reverse proxy and will write logs to the current directory:

The easiest way to run nginx is using docker:

We configured the requests to be forwarded to a backend server on port 8081. Note, that if you’re doing so on localhost use host.docker.internal, like we did above.

Now, we are going to call our server and see if it creates an access-log:

If you didn’t create a service that listens on localhost:8081 it will return 502 Bad Gateway. What is more interesting is to look at the new log file nginx-access.log in current-directory? Does it contain your request? If so, great, let’s move on :)

Next, we are going to configure our access_log to be sent to our UDP server by modifying nginx config file and rerunning nginx server (more data regarding configuring logging you can find here)

After running our Go UDP server, let’s call /hello again using curl and this time we should see that the access log is written as a Go log :)

Create a custom log format

Now, let’s change the format of the access-log to json so it’ll be easier to parse it on the Go server:

As you can see added a log_format definition name tufin and we are using it in the server’s access_log. You can find out more here.

Parse Access-Log sent from Nginx

Notes:

  • Nginx adds some prefix before the json that we defined in the previous section, that is why we start by getting request substring json
  • We have to trim “\x00” otherwise it fail to unmarshal json

You can find the full code in GitHub :)

From the Security Policy Company. This blog is dedicated to cloud-native topics such as Kubernetes, cloud security and micro-services.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store