Collecting logs using Go

Author: Alexander Tryapkin, DevOps at Hostkey

In this article, I would like to share my experience in solving issues with collecting logs using Go. As a DevOps beginner, I have chosen the Go programming language to learn and solve work tasks.The syslog library is available for sending syslog logs, alas it is not suitable in our case because this package is unavailable on Windows, while our task is to make a multiplatform sender of system installation logs to a remote syslog server. Furthermore, we need to send the logs in a custom format, namely in json, in order to simplify their subsequent processing. At the same time, it is important that the program runs the same on Linux and on Windows, that it does not require installation, and that it performs its task and can be removed from the system. In short, you have to reinvent the wheel. Let's proceed.

We will use syslog-ng as the receiving end. Let's consider the parameters which we are interested in collecting logs. Indeed, how we will send them depends on the parameter specifications.

Firstly, we specify a new source for receiving logs from remote servers, and there are options here - depending on our needs, we can collect logs via UDP, TCP, or also we can use TLS for encryption and authentication. The most interesting option is TLS, but we will take a quick look at the other methods, from the simplest to the more complicated.

1) UDP. To collect logs over UDP, you will need the following parameters in your syslog-ng configuration:

source s_network {
	network( ip("0.0.0.0") #IP, to which logs are received, 0.0.0.0 - for all
						 transport("udp")  );       };

The default port is 514/UDP. The documentation warns us of the need to increase the UDP buffer if the logs are to be sent at a high rate; otherwise some messages may be lost. In case of packet loss, the logs will also be lost, so obviously this is far from ideal.

2) TCP. This option avoids the above-mentioned problems and, according to the documentation, it is by default. Here is an example config:

source s_network {
	network( ip("0.0.0.0") ); };

3) TLS. To use this protocol, you need to configure the server. The official documentation has quite detailed step-by-step instructions. For example:

source s_remote_tls { 
network ( ip ("0.0.0.0") port(6514) 
transport("tls") 
tls( key-file("/etc/syslog-ng/cert.d/serverkey.pem") cert-file("/etc/syslog-ng/cert.d/servercert.pem")
ca-dir("/etc/syslog-ng/ca.d")
peer-verify(yes)) ); };

With this configuration option, we will only accept logs from authenticated clients. In other words, the client uses a valid certificate and the logs come from the IP address or domain name under which the certificate is issued. If there is no task to authenticate users, then you can specify the peer-verify as (no) and then only apply encryption.

After considering a number of options, we decided to create a small program that will send our logs.

First, let's figure out how to send a syslog message to the server so that it accepts and processes it. From the documentation, we see that received messages must comply with the RFC3164 or RFC5424 protocols. However, as this is not the final version, let's just try to send a log using RFC3164, which looks like this:

<30>Dec 25 21:55:36 19202.example.ru systemd[1]: Starting Cleanup of Temporary Directories...

Now let's review what each part of the message means:

• <30> - a header containing information about severity and facility. The encoded information can be decrypted using a table, in this case it contains facility - system and severity - info.
• Dec 25 21:55:36 — timestamp.
• 19202.example.ru — hostname.
• Systemd[1]: — message tag indicating which program sent the message.
• Starting Cleanup of Temporary Directories… — the message itself.

Let's try to send a message in this format to our test syslog-ng server configured to receive logs via UDP. For this we will use the net library:

	logsrv, err := net.ResolveUDPAddr("udp4", "141.105.70.24:514")
	if err != nil {
		log.Fatal(err)
	}
	logwriter, err := net.DialUDP("udp4", nil, logsrv)
	if err != nil {
		log.Fatal(err)
	}
	defer logwriter.Close()
	_, err = logwriter.Write([]byte("<30>Dec 25 21:55:36 test-host go-logger: Hello Habr!"))
	_, err = logwriter.Write([]byte("<30>Dec 25 21:55:36 test-host go-logger: This is test go-logger!"))
	if err != nil {
		log.Fatal(err)
	}

After executing the code, we see that the server has received our messages and processed them. The messages are written to the specified file:

[root@19181 ~]# cat /var/log/test 
Dec 25 21:55:36 test-host go-logger: Hello Habr! 
Dec 25 21:55:36 test-host go-logger: This is test go-logger!

Now let's send the logs over TCP. We reconfigure the server to receive logs via TCP and try:

tcpAddr, err := net.ResolveTCPAddr("tcp", "141.105.70.24:514")
	if err != nil {
		log.Fatal(err)
	}
	logwriter, err := net.DialTCP("tcp", nil, tcpAddr)
	if err != nil {
		log.Fatal(err)
	}
	defer logwriter.Close()
	_, err = logwriter.Write([]byte("<30>Dec 25 21:55:36 test-host go-logger: Hello Habr!"))
	_, err = logwriter.Write([]byte("<30>Dec 25 21:55:36 test-host go-logger: This is test go-logger!"))
	if err != nil {
		log.Fatal(err)
	}

Dec 25 21:55:36 test-host go-logger: Hello Habr!<30>Dec 25 21:55:36 test-host go-logger: This is test go-logger!

But what we see: something went wrong, two messages are merged into one, and the second message is not parsed. When we sent logs over UDP, this problem did not arise, since each message goes in its own packet and is processed separately. The solution is actually simple - I missed that each message must end with a line break \n. We edit and try:

	_, err = logwriter.Write([]byte("<30>Dec 25 21:55:36 test-host go-logger: Hello Habr!\n"))
	_, err = logwriter.Write([]byte("<30>Dec 25 21:55:36 test-host go-logger: This is test go-logger!\n"))

[root@19181 ~]# cat /var/log/test 
Dec 25 21:55:36 test-host go-logger: Hello Habr! 
Dec 25 21:55:36 test-host go-logger: This is test go-logger!

Now everything is ok!

We have figured out how to send a message. Now, it's time to apply this knowledge. Keep in mind that if we want to send messages in json format (this will greatly facilitate the task of processing the logs in the future), we need to disable parsing in syslog-ng. To do this, just add flags (no-parse) to the source. Next, we will try to send the logs via the TLS protocol and in json format already in the form of a full-fledged program:

package main

import (
	"bufio"
	"crypto/tls"
	"crypto/x509"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"time"

	"github.com/pborman/getopt/v2"
)
	
// We set the structure of our message, here we are not limited to syslog, we send only what we need or, conversely, add:
type message struct {
	Time     string `json:"timestamp"`
	Hostname string `json:"host"`
	Programm string `json:"programm"`
	Body     string `json:"message"`
}
	
func main() { 
	// We will pass the required parameters to our program by means of keys, and the getopt library will help us with this.
	optSyslogSrv := getopt.StringLong("dest", 'd', "", "Remote syslog server with port ip:port, required")
	optReadFromFile := getopt.StringLong("file", 'f', "", "Read log from file")
	optProg := getopt.StringLong("prog", 'p', "go-logger", "Programm tag, optional,  default - go-logger")
	optHost := getopt.StringLong("host", 'H', "", "Host override")
	optHelp := getopt.BoolLong("help", 'h', "Display usage")
	optVerb := getopt.BoolLong("verbose", 'v', "Display outgoing msgs")
	optCa := getopt.StringLong("ca", 'c', "cacert.pem", "CA")
	optCert := getopt.StringLong("cert", 'C', "clientcert.pem", "Cert")
	optKey := getopt.StringLong("key", 'K', "", "clientkey.pem", "Key")

	getopt.Parse()
	// If the program is run with the -h --help key, or if the required parameter is not given, a usage hint will be displayed:
	if *optHelp || len(*optSyslogSrv) == 0 {
		getopt.Usage()
		os.Exit(0)
	}

	var hostname string
	var scanner *bufio.Scanner

	if len(*optHost) != 0 { // If hostname is specified by a key, take the information from there.
		hostname = *optHost
	} else { // Otherwise, we can get it from the system:
		hostname, _ = os.Hostname()
	}
	// Our program can either take the logs from stdin while running in the pipeline "anyscript.sh | go-logger -d 127.0.0.1:514", or from the file. If the parameter is specified, then we take it from the file:
	if len(*optReadFromFile) != 0 {
		file, err := os.Open(*optReadFromFile) // Open the file
		if err != nil {
			log.Fatal(err)
		}
		defer file.Close() // Schedule the file to close when finished
		scanner = bufio.NewScanner(file) // Read the file
	} else {
		scanner = bufio.NewScanner(os.Stdin) // Read stdin
	}

	msg := message{Hostname: hostname, Programm: *optProg} // Enter the data into the structure
	// The TLS part of sending our logs:
	caCert, _ := ioutil.ReadFile(*optCa) // Load the CA server from a file
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)

	cert, err := tls.LoadX509KeyPair(*optCert, *optKey) // Load the certificate and the client private key from the files.
	if err != nil {
		log.Fatal(err)
	}
	tlsConf := &tls.Config{ // Creating a TLS configuration
		RootCAs:      caCertPool,
		Certificates: []tls.Certificate{cert},
	}

	logwriter, err := tls.Dial("tcp", *optSyslogSrv, tlsConf) // Setting up a TLS connection
	if err != nil {
		log.Fatal(err)
	}
	defer logwriter.Close() // Schedule the termination of the connection when it is over

	for scanner.Scan() { // Process each message received by the scanner
		sendMsg := message{
			Time:     time.Now().Format("2006-01-02T15:04:05.00-07:00"), // Time in the format we need
			Body:     scanner.Text(),                                    //Message
			Hostname: msg.Hostname,
			Programm: msg.Programm,
		}
		data, err := json.Marshal(sendMsg) //Marshaling our json
		if err != nil {
			log.Fatal(err)
		}
		if *optVerb { // If the -v parameter is given, we print the message to be sent
			fmt.Println(string(data))
		}
		_, err = logwriter.Write(append(data, "\n"...)) // Send the message by adding a line break
		if err != nil {
			log.Fatal(err)
		}
	}
}

Now, we can try to execute the program by sending the same two lines, and the server receive our logs:

{"timestamp":"2022-12-26T00:19:23.54+03:00","host":"test-go-logger","programm":"go-logger","message":"Hello habr!"}
{"timestamp":"2022-12-26T00:19:23.54+03:00","host":"test-go-logger","programm":"go-logger","message":"Lets test!"}

This program is one of the first ones I wrote in Go. In doing so, I figured out how the syslog protocol works and mastered the basics of a new programming language for me. The program made it possible to unify the sending of logs on different operating systems, regardless of the family, in places where it is not possible to use syslog-ng. Currently, we are adapting this newly-created program for further use within the infrastructure of our company.