Posted   by 

Docker Syslog

Using Seq.Input.Syslog, Seq is able to ingest syslog messages — both RFC3164 and RFC5424 formats — as structured logs.

Contents

Connect Syslog & Docker. Syslog lets you monitor and review device logs across your entire infrastructure. StrongDM uses standardized messaging protocols like Syslog to send access and session logs from Docker containers to your logging tools. Free 14-day trial, no credit card required. We utilise the docker manifest for multi-platform awareness. More information is available from docker here and our announcement here. Simply pulling ghcr.io/linuxserver/syslog-ng should retrieve the correct image for your arch, but you can also pull specific arch images via tags. The architectures supported by this image are: Architecture. Specialised container to build doc for the rsyslog.com website. By rsyslog. Updated 19 days ago. A ready-to run appliance for rsyslog doc management. For Seq to ingest syslog messages, you can deploy datalust/seq-input-syslog as a Docker container alongside a separate Seq instance. The seq-input-syslog container receives syslog messages (via UDP on port 514 by default), and forwards them to the Seq ingestion endpoint specified in the SEQADDRESS environment variable.

What is syslog?
Syslog message formats
RFC 3164
RFC 5424
How to ingest syslog messages into Seq
Method 1: (Windows, Docker) installing Seq.Input.Syslog directly in Seq
Method 2: (Docker) running a separate seq-input-syslog 'sidecar' container
Example usage: analysing NGINX logs with Seq

What is syslog?

Syslog (System Logging) is a logging format and protocol created by Eric Allman as part of Sendmail in the 1980s, and has since gained popularity in *nix based systems — including BSD (Berkeley Software Distrubution) Unix, Linux, and macOS — as well as network devices, such as printers and routers.

Syslog was first standardized by the IETF (Internet Engineering Task Force) in 2001, when the team published a Request for Comments titled 'The BSD Syslog Protocol' (RFC 3164). 'The Syslog Protocol' (RFC 5424), a more modern syslog standard, was later published in 2009, and obsoleted RFC 3164.

Seq.Input.Syslog is able to parse message formats described in both RFC 3164 and RFC 5424, with a few important things to note.

Firstly, Seq.Input.Syslog currently only supports receiving syslog messages over UDP. Secondly, the MSG component of syslog messages sent to Seq via Seq.Input.Syslog are not currently parsed — even if they contain structured elements, they are sent to Seq as free text.

Finally, we do not recommend using Seq.Input.Syslog as a first choice for most use-cases, as there are more convenient log formats that also add structure to the MSG portion of the log. We hope syslog support does help those who maintain and support systems that rely on syslog.

Syslog message formats

Even though RFC 3164 has been obsoleted by RFC 5424, the older log format is still supported in many applications. Seq.Input.Syslog supports structured events for both versions.

Here is a handy reference for both log formats.

Docker Syslog

RFC 3164

Take the following RFC 3164-formatted syslog message

This message is made up of several important 'parts'.

Below is our simplified explanation of Section 4.1 syslog Message Parts in RFC 3164.

  • PRI — or 'priority', is a number calculated from Facility (what kind of message) code and Severity (how urgent is the message) code: PRI = Facility * 8 + Severity
  • TIMESTAMP — format is Mmm dd hh:mm:ss
  • HOSTNAME — must contain the hostname, IPv4 or IPv6 address of the message sender
  • MSG is made up of two parts:
    • TAG — the name of the program or process that generated the message. Usually followed by a : or a [pid]: (the beginning of the MSG CONTENT)
    • CONTENT — the details of the message

These parts are parsed into structured log messages in Seq using the Seq syslog input. Here's what the above message looks like in Seq:

RFC 5424

Here is an example RFC 5424-formatted syslog message

Docker Syslog Inside Container

Again, this message is made up of important 'parts'. These parts are explained in the next section.

RFC 5424 messages contain more parts than RFC 3164, probably due to no longer being limited to maximum 1024 byte message size.

Syslog

This is our simplified explanation of Section 6. Syslog Message Format in RFC 5424.

  • HEADER
    • PRI — or 'priority', is a number calculated from Facility (what kind of message) code and Severity (how urgent is the message) code: PRI = Facility * 8 + Severity
    • VERSION — version is always '1' for RFC 5424
    • TIMESTAMP — valid timestamp examples (must follow ISO 8601 format with uppercase 'T' and 'Z')
      • 1985-04-12T23:20:50.52Z
      • 2003-08-24T05:14:15.000003-07:00
      • - ('nil' value) if time not available
    • HOSTNAME — using FQDN (fully qualified domain name) is recommended, e.g. mymachine.example.com
    • APP-NAME — usually the name of the device or application that provided the message
    • PROCID — often used to provide the process name or process ID (is - 'nil' in the example)
    • MSGID — should identify the type of message, more detail in RFC 5424 Section 6.2.7. MSGID
  • STRUCTURED-DATA — named lists of key-value pairs for easy parsing and searching, more detail in RFC 5424 Section 6.3. STRUCTURED-DATA
  • MSG — details about the event
    • if the MSG is encoded in UTF-8, the string must start with the Unicode byte order mask (BOM), more detail in RFC 5424 Section 6.4. MSG

And here's what the same RFC 5424 formatted message looks like in Seq:

How to ingest syslog messages into Seq

There are two ways to send syslog messages to Seq:

  • by installing Seq.Input.Syslog directly in Seq (Windows and Docker), or
  • by running a separate seq-input-syslog (Docker only).

Below is a convenient shell command to test if your setup is working, while you're getting set up with Seq.Input.Syslog, you can use this test syslog message to check everything is configured correctly (if your system has netcat):

Docker Syslog

Method 1: (Windows, Docker) installing Seq.Input.Syslog directly in Seq

The simplest method is

  1. installing Seq.Input.Syslog directly in Seq via Settings > Apps, and then
  2. set up an instance (i.e. syslog receiver) using the Add Instance button in Apps

Docker Syslog Setup

If you are running Seq as a Windows service, you must first check that your chosen syslog listener port is allowed through Windows firewall (UDP port 514 is the default, but you can pick a different UDP port).

If you are running Seq in Docker, you must first expose your chosen syslog listener port via your docker run command or docker-compose file. Remember, this is only required if you are installing Seq.Input.Syslog in Settings > Apps. Here's a docker run datalust/seq command with the correct ports exposed:

If you are running Seq in Docker, we recommend you choose the seq-input-syslog sidecar container method, where you do not need to expose any extra ports on the seq container, and will also save you the extra app installation steps.

Method 2: (Docker) running a separate seq-input-syslog 'sidecar' container

For Seq to ingest syslog messages, you can deploy datalust/seq-input-syslog as a Docker container alongside a separate Seq instance.

The seq-input-syslog container receives syslog messages (via UDP on port 514 by default), and forwards them to the Seq ingestion endpoint specified in the SEQ_ADDRESS environment variable.

Here's a docker run datalust/seq-input-syslog command with the default UDP listener port exposed, and sends logs to a Seq ingestion endpoint at https://seq.example.com:5341:

Example: analysing NGINX logs with Seq

Here is an example docker-compose.yml which uses Docker's syslog log driver to forward NGINX docker container logs to Seq (i.e. whatever you see in stdout when you run docker logs -f <container-name>).

Important note: This is not our recommended way to get NGINX logs into Seq. Instead, use Seq.Input.GELF.

Why is localhost allowed for the logging-driver syslog-address? This is because the logging driver daemon is actually on the host machine, and localhost is resolved outside of the docker container.

Here is what an NGINX log looks like in Seq, after accessing localhost:8888:

That's it! Hope you're up and running in minutes with Seq as your new centralized syslog server :)

If you have any feedback or would like to contribute, please create a GitHub issue for Seq.Input.Syslog.

Happy logging! ❤️

Building services using Spring Boot gives a development team a jump start on many production concerns, including logging. But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space.

The Docker logging drivers capture all the output from a container’s stdout/stderr, and can send a container’s logs directly to most major logging solutions (syslog, Logstash, gelf, fluentd).

As an added benefit, by making the logging implementation a runtime choice for the container, it provides flexibility to use a simpler implementation during development but a highly-available, scalable logging solution in production.

If you would rather explore an alternate Docker logging architecture, using a dedicated container that senses all other containers and routes their log events, see my article on Docker logspout.

Spring Boot logging to the console

Using our Spring Boot example fully described here, we will have it output DEBUG level logs to the console. It uses SLF4J/Logback so we configure ‘src/main/resources/logback.xml’ to send events to its ConsoleAppender.

Later in this article we put this application into a Docker container, but for now let’s run it directly on our Ubuntu host machine.

[Shell]
$ sudo apt-get install curl git openjdk-7-jdk maven -y
$ git clone https://github.com/fabianlee/gs-rest-service.git
$ cd gs-rest/service/complete
$ ./mvnw package
$ ufw allow 8080/tcp
$ java -jar target/gs-rest-service-0.1.0.jar
[/Shell]

Then you can make a client call into this service by using curl:

[Shell]
$ curl http://localhost:8080/greeting
[/Shell]

Each call to the service will output debug level logs to the console. This console level output will be redirected using the appropriate Docker logging driver in the subsequent sections.

Spring Boot project deployed in Docker container

Now that we have seen our example project deployed directly on our host machine, let’s move on to deploying it into a Docker container. If you have not installed Docker, here are instructions for Ubuntu.

If you want to build the Docker image and run gs-rest-service in a docker container where port 8080 is mapped directly to the docker host:

[Shell]
$ sudo docker build -t gs-rest-service -f src/main/docker/Dockerfile .
$ sudo docker run -p 8080:8080 gs-rest-service
[/Shell]

We have the container’s port 8080 mapped directly to the host, so you can call curl on localhost:8080 exactly like before.

Docker container sending logs to json-file

Break out of the container started earlier, and start the container again explicitly specifying a json-file logging driver.

[Shell]
$ sudo docker run -p 8080:8080 –log-driver=json-file gs-rest-service
[/Shell]

Now you can view the latest logs and check the path of the persisted json file by using:

[Shell]
$ sudo docker logs $(docker ps -q)
$ sudo docker inspect –format='{{.LogPath}}’ $(docker ps -q)
[/Shell]

Note: This .json file is located on the Docker host, not the container.

Docker container sending logs to syslog

Before we send syslog output to a host, let’s make sure the host either has rsyslog enabled (see my article here for Ubuntu), or we use netcat on the host to echo what comes into port 514 via UDP (use sudo or root because port 514 < 1024).

[Shell]

# ufw allow 514/udp

# while true; do { nc -vlu 514; } done

[/Shell]

Now break out of the Docker container started earlier, and start the container again explicitly specifying a syslog logging driver to the host where you have enabled the syslog listener on 514/udp.

[Shell]
$ sudo docker run -p 8080:8080 –log-driver=syslog –log-opt syslog-address=udp://192.168.1.2:514 –log-opt syslog-facility=daemon –log-opt syslog-format=rfc5424 –log-opt tag=”{{.ImageName}}/{{.ID}}” gs-rest-service
[/Shell]

The ‘Docker logs’ command does not have the ability to show you these network logs, but you should have received logs from netcat or ‘/var/log/syslog’ that looks like:

[RAW]

<30>1 2017-03-20T22:00:00-05:00 hpi5 gs-rest-service/78eab1df792a 26022 gs-rest-service/78eab1df792a 03:00:00.806 [http-nio-8080-exec-3] DEBUG hello.GreetingController – debug message slf4j
<30>1 2017-03-20T22:00:00-05:00 hpi5 gs-rest-service/78eab1df792a 26022 gs-rest-service/78eab1df792a 03:00:00.807 [http-nio-8080-exec-3] INFO hello.GreetingController – info message slf4j

[/RAW]

Summary

The overarching idea is that these container log messages are now being directed by the person deploying the Docker container, and not the developer who packages the application.

This loose coupling makes the logging implementation an orthogonal concern that can be centrally operated for production environments (e.g. ELK), yet simple to debug during the development lifecycle.

REFERENCES

https://www.loggly.com/blog/top-5-docker-logging-methods-to-fit-your-container-deployment-strategy/

docker run -it ubuntu:trusty /bin/bash

docker ps -q (last container id)

docker ps -a (all container id, even stopped)

docker inspect -f ‘{{.Name}} – {{.NetworkSettings.IPAddress }}’ $(docker ps -q)

docker exec -it $(docker ps -q) bash (get to shell of latest container)

logger -p local0.warn -d -n myhost “test message to catchall” -u /ignore/socket

docker inspect –format='{{.LogPath}}’ $(docker ps -q)