What are Linux logs?
Linux logs are a critical component of the OS that gives a snapshot of everything that has been happening in the system. It helps in tracking events within the system such as process initiation/failure, errors in running applications, login failures, boot-up process etc.
They are essential for debugging running applications, tracking security events, such as a brute force attack etc.
The log files also provide important diagnostic information in the even of some kind of failure, so that system administrators and gather information and fix the system.
For example a server application like apache or nginx or postfix failed to start or suddenly stopped working, then it is always best to first check the log files to see what went wrong. The log files will most of the time contain accurate information about the cause of the error which will help in fixing the problem.
In this article we take a quick look at some of the common log files found on linux systems. They would be present on both server and desktop linux installations.
Types of log files
On linux there are lots of log files and some common ones include:
- /var/log/syslog
- /var/log/auth.log
- /var/log/daemon.log
- /var/log/kern.log
Logs like syslog, kern.log and auth.log are found on almost all linux distros.
Before looking at the types of log files, let's see the location of Linux logs under "/var/log".
Navigate to the directory and list the files.
cd /var/log/
$ ls alternatives.log btmp dmesg.1.gz gdm3 kern.log speech-dispatcher vmware-network.1.log vmware-vmsvc-root.1.log wtmp apt cups dmesg.2.gz gpu-manager.log lastlog syslog vmware-network.2.log vmware-vmsvc-root.2.log auth.log dist-upgrade dpkg.log hp openvpn ubuntu-advantage.log vmware-network.3.log vmware-vmsvc-root.3.log boot.log dmesg faillog installer private ubuntu-advantage-timer.log vmware-network.4.log vmware-vmsvc-root.log bootstrap.log dmesg.0 fontconfig.log journal README unattended-upgrades vmware-network.log vmware-vmtoolsd-root.log
In the above screenshot, we can see the log files present in the system. For this tutorial, I will be using 'Ubuntu 22.10'.
On a different system the contents of /var/log looks like this:
enlightened@enlightened:~$ ls /var/log/ alternatives.log apache2 boot.log dmesg.4.gz dpkg.log.9.gz kern.log.3.gz mail.log.4.gz syslog.6.gz ubuntu-advantage-timer.log.2.gz vbox-setup.log.1 alternatives.log.1 apport.log boot.log.1 dpkg.log faillog kern.log.4.gz mysql syslog.7.gz ubuntu-advantage-timer.log.3.gz vbox-setup.log.2 alternatives.log.10.gz apport.log.1 bootstrap.log dpkg.log.1 firebird lastlog prime-offload.log sysstat ubuntu-advantage-timer.log.4.gz vbox-setup.log.3 alternatives.log.11.gz apport.log.2.gz btmp dpkg.log.10.gz firewalld lxc prime-supported.log tallylog ubuntu-advantage-timer.log.5.gz vbox-setup.log.4 alternatives.log.12.gz apport.log.3.gz btmp.1 dpkg.log.11.gz fontconfig.log mail.err private ubuntu-advantage.log ubuntu-advantage-timer.log.6.gz wpslog alternatives.log.2.gz apt cups dpkg.log.12.gz gpu-manager.log mail.err.1 README ubuntu-advantage.log.1 ufw.log wtmp alternatives.log.3.gz aptitude distccd.log dpkg.log.2.gz gufw.log mail.err.2.gz samba ubuntu-advantage.log.2.gz ufw.log.1 wtmp.1 alternatives.log.4.gz aptitude.1.gz dist-upgrade dpkg.log.3.gz hp mail.err.3.gz sddm.log ubuntu-advantage.log.3.gz ufw.log.2.gz Xorg.0.log alternatives.log.5.gz auth.log dmesg dpkg.log.4.gz installer mail.err.4.gz syslog ubuntu-advantage.log.4.gz ufw.log.3.gz Xorg.0.log.old alternatives.log.6.gz auth.log.1 dmesg.0 dpkg.log.5.gz journal mail.log syslog.1 ubuntu-advantage.log.5.gz ufw.log.4.gz Xorg.1.log alternatives.log.7.gz auth.log.2.gz dmesg.1.gz dpkg.log.6.gz kern.log mail.log.1 syslog.2.gz ubuntu-advantage.log.6.gz unattended-upgrades Xorg.pid-12656.log alternatives.log.8.gz auth.log.3.gz dmesg.2.gz dpkg.log.7.gz kern.log.1 mail.log.2.gz syslog.3.gz ubuntu-advantage-timer.log upgrade alternatives.log.9.gz auth.log.4.gz dmesg.3.gz dpkg.log.8.gz kern.log.2.gz mail.log.3.gz syslog.4.gz ubuntu-advantage-timer.log.1 vbox-setup.log enlightened@enlightened:~$
Following are the types of log files that will be discussed in this section:
- System logs
- Application logs
- Non-human readable logs
- Syslog
Even though there are several logs files in linux, we can classify them into 4 major categories:
1. Syslog (/var/log/syslog)
Syslog is a standard network-based logging protocol which can be configured to store logs for various devices and applications to a centralized server.
Ubuntu uses rsyslog for syslog configuration which is discussed in the later section. It stores all kinds of messages except "auth" related messages.
The Syslog daemon listens for logs and writes them to a specific location. The location(s) is defined in the configuration file for the daemon. rsyslog is the Syslog daemon shipped with most of the distros.
You can view the local syslog on your system under "/var/log/syslog".
tail -2f /var/log/syslog Mar 19 13:06:18 overfittedropout-virtual-machine NetworkManager[853]: [1679211378.7842] manager: NetworkManager state is now CONNECTED_GLOBAL Mar 19 13:06:28 overfittedropout-virtual-machine systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully. Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Activating service name='org.gnome.Nautilus' requested by ':1.24' (uid=1000 pid=1292 comm="/usr/bin/gnome-shell" label="unconfined") Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Successfully activated service 'org.gnome.Nautilus'
In the above excerpt, we can see Nautilus service is loaded, which indicates that the file system is accessed/interacted with.
2. dmesg log (/var/log/dmesg)
The /var/log/dmesg log file is used to write the kernel messages retrieved from the kernel ring buffer. The kernel ring buffer is a circular buffer that is the first data structure storing log messages when the system boots up.
This particular log file contains a lot of information about the hardware detection process. For example whenever you plug a usb device into the system an entry shall appear in the dmesg log file.
One way of viewing the dmesg log file is by simply opening the /var/log/dmesg file. Another way is to use the dmesg command which presents the log file in a much easier to read format.
You can view more options to be used with dmesg –help. Here is a quick and simple example:
$ dmesg [ 0.000000] Linux version 5.19.0-31-generic (buildd@lcy02-amd64-087) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.2.0-3ubuntu1) 12.2.0, GNU ld (GNU Binutils for Ubuntu) 2.39) #32-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 20 15:20:08 UTC 2023 (Ubuntu 5.19.0-31.32-generic 5.19.17) [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.19.0-31-generic root=UUID=01276451-a735-47ce-96df-1d22f6b2654e ro find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US quiet splash [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Hygon HygonGenuine [ 0.000000] Centaur CentaurHauls [ 0.000000] zhaoxin Shanghai
In the above excerpt, we can see the boot-up logs where Linux is loading the boot image and also printing the supported CPU preference for the kernel. We have written a full article on how to use the dmesg command to view kernel logs over here:
https://www.binarytides.com/dmesg-command-examples-in-linux/3. Authorization log (/var/log/auth.log)
– Keeps track of login for users. It also logs privilege escalation events such as sudo.
$ sudo head -n6 /var/log/auth.log Feb 22 20:47:35 <strong>overfittedropout-virtual-machine</strong> systemd-logind[749]: New seat seat0. Feb 22 20:47:35 overfittedropout-virtual-machine systemd-logind[749]: Watching system buttons on /dev/input/event0 (Power Button) Feb 22 20:47:35 overfittedropout-virtual-machine systemd-logind[749]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard) Feb 22 20:47:47 overfittedropout-virtual-machine gdm-autologin]: gkr-pam: no password is available for user <strong>Feb 22 20:47:47 overfittedropout-virtual-machine gdm-autologin]: pam_env(gdm-autologin:session): deprecated reading of user environment enabled Feb 22 20:47:47 overfittedropout-virtual-machine gdm-autologin]: pam_unix(gdm-autologin:session): session opened for user overfittedropout(uid=1000) by (uid=0)</strong>
In the above excerpt, we can see logon events for user overfittedropout. We can also see that the user has enabled auto-login.
4. Daemon Log (/var/log/daemon.log)
- Daemons are programs that run in the background, usually without user interaction. Daemon log stores information on running processes.
$ tail -n 20 /var/log/daemon.log Feb 23 23:03:34 overfittedropout-virtual-machine systemd[1014]: Started Application launched by gnome-session-binary. Feb 23 23:04:01 overfittedropout-virtual-machine dbus-daemon[1057]: [session uid=1000 pid=1057] Activating service name='org.gnome.TextEditor' requested by ':1.73' (uid=1000 pid=1494 comm="/usr/bin/nautilus --gapplication-service" label="unconfined")
In the above log excerpt, we can see the gnome text editor was opened.
5. Kernel log (/var/log/kern.log)
Logs information from the linux kernel.
$ tail -n20 kern.log Feb 23 23:22:26 overfittedropout-virtual-machine kernel: [ 213.623243] audit: type=1400 audit(1677174746.121:56): apparmor="ALLOWED" operation="connect" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2192 comm="soffice.bin" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000 Feb 23 23:22:26 overfittedropout-virtual-machine kernel: [ 213.623489] audit: type=1400 audit(1677174746.121:57): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2192 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 Feb 23 23:22:26 overfittedropout-virtual-machine kernel: [ 213.623492] audit: type=1400 audit(1677174746.121:58): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2192 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 Feb 23 23:22:29 overfittedropout-virtual-machine kernel: [ 217.180515] audit: type=1400 audit(1677174749.677:59): apparmor="ALLOWED" operation="connect" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2213 comm="soffice.bin" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000 Feb 23 23:22:29 overfittedropout-virtual-machine kernel: [ 217.180730] audit: type=1400 audit(1677174749.677:60): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2213 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 Feb 23 23:22:29 overfittedropout-virtual-machine kernel: [217.180733] audit: type=1400 audit(1677174749.677:61): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2213 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
In the above excerpt, we can see libre-office writers getting loaded.
6. Login failure logs(var/log/faillog)
Contains login failure events of the system. The "faillog" command is used to view the log file contents. This log is stored in binary format and cannot be read as plain text.
$ faillog -a Login Failures Maximum Latest On root 0 0 01/01/70 05:30:00 +0530
You can view additional options and syntax using "man faillog" and "faillog –help".
7. Current login logs(/var/log/wtmp)
Contains login info used by other utilities to find out who's logged in. To view currently logged in users, use the "who" command.
who overfittedropout tty2 2023-02-24 01:04 (tty2)
In the above excerpt, we can see the user "overfittedropout" is logged into the system since 01:04.
Note:
Most Linux files have time stamps which are quintessential to establish timelines of incident/errors and helps us filter the logs. It is a good practice to keep a consistent time zone configuration for all the servers.
Here is an exhaustive list of log files in Linux:-
LinuxLogFiles - Community Help Wiki (ubuntu.com)
Application Specific Logs
Different applications like apache, nginx, postfix also create their own specific log files that a lot contain runtime information useful for debugging issues.
For example if mail delivery has failed you might want to check the postfix log files or if some webpage is showing 404 not found or 500 internal server error then you have to check the error logs of the http server application whether apache or nginx.
1. Apache Error Log
$ cat /var/log/apache2/error.log [Fri Mar 31 08:45:40.112067 2023] [mpm_prefork:notice] [pid 1294] AH00163: Apache/2.4.54 (Ubuntu) configured -- resuming normal operations [Fri Mar 31 08:45:40.112092 2023] [core:notice] [pid 1294] AH00094: Command line: '/usr/sbin/apache2' $
2. Apache Access Log
The access log contains an entry for every http request made to the server from any client. It can be viewed like this:
$ cat /var/log/apache2/access.log
3. Apache tomcat logs(apache-tomcat-10.1.5/logs/catalina.out)
These are logs from apache tomcat server installation.
$ head -n 2 catalina.out ./catalina.sh: 1: eval: /home/overfittedropout/Downloads/openlogic-openjdk-8u362-b09-linux-x32/bin/bin/java: not found
In the above log, we can see 'java' is not found which can be used to debug any issues related to the tomcat server.
Commands to view log files
Log files on linux are mostly in plain text format with each entry as a newline. Some commonly used commands to view log files are:
less – Lets you page through a text file, displaying a screenful of text each time
grep – Find all occurrences of a search term in a file or filter a log file
tail/head - Output the last few lines of files
Syntax and usage
1. less
The less command pipes the output and makes it scrollable in the terminal. It can be used for viewing any log file in text format. For example we can view the syslog file like this:
$ cat /var/log/syslog | less Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Starting Flush Journal to Persistent Storage... Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Set the console keyboard layout. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load/Save Random Seed. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes). Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: modprobe@pstore_zone.service: Deactivated successfully.
The cat command prints the whole file to "less" which in turn makes it scrollable and easy to view.
Searching terms using 'less'
For searching specific terms in a file, press the forward (/) button while you are viewing the file, write the search term and hit enter.
Syntax: /
For example - /2022-10-13 will search for logs which are dated on 13th October and highlight it.
You can view the options of less using "less --help" and "man less".
Note: To search backward from your current position in the file toward the start of the file, press the "?" key and type your search term. To find the next matching item, press "n". To search for the previous matching item, press "N".
Press 'q' to exit less.
2. grep
The Linux grep command is a string and pattern matching utility that displays matching lines from specified files. We can also use it in integration with other commands by piping the output.
Syntax – | grep
$ cat /etc/passwd | grep overfittedropout overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash
In the above output, we can see grep has filtered the output of "cat /etc/passwd/" and displayed the lines which contains the word "overfittedropout".
The same output can also be obtained by "grep overfittedropout /etc/passed".
Syntax – grep
$ grep overfittedropout /etc/passwd overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash
Note:
The above command is case-sensitive.
In case we forget the proper case of a keyword we are searching, we can view all the lines using the following command.
Syntax – grep -i
The "-i" flag denotes ignore case.
$ cat /etc/passwd |grep -i OverfittedRopouT overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash
By default, grep searches for the target that appears anywhere in that line, including inside another string. We can force grep to search for a keyword as a whole using the -w flag.
Syntax – grep -w
Invert can be performed using "-v" flag searches using grep in the following way.
Syntax – grep -v
The -v represents inverse search.
Let's say we want to search a line in '/etc/passwd/', where the keyword "no login" is not present.
$ cat /etc/passwd | grep -v nologin root:x:0:0:root:/root:/bin/bash sync:x:4:65534:sync:/bin:/bin/sync tss:x:106:113:TPM software stack,,,:/var/lib/tpm:/bin/false whoopsie:x:117:124::/nonexistent:/bin/false speech-dispatcher:x:119:29:Speech Dispatcher,,,:/run/speech-dispatcher:/bin/false gdm:x:125:133:Gnome Display Manager:/var/lib/gdm3:/bin/false hplip:x:126:7:HPLIP system user,,,:/run/hplip:/bin/false gnome-initial-setup:x:127:65534::/run/gnome-initial-setup/:/bin/false overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash
You can view additional options and syntax using "grep --help" and "man grep".
3. tail
As the name suggests, the tail command prints out lines beginning from the end of the file. It is complementary to the "head" command which prints text from the beginning of the file.
cat /var/log/syslog | head Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Starting Flush Journal to Persistent Storage... Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Set the console keyboard layout. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load/Save Random Seed. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes). Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: modprobe@pstore_zone.service: Deactivated successfully. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load Kernel Module pstore_zone. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: modprobe@pstore_blk.service: Deactivated successfully. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load Kernel Module pstore_blk. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: [email protected]: Deactivated successfully. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load Kernel Module drm.
You can specify the number of lines that need to be printed either from top or bottom with "head" and "tail" commands respectively with the -n flag.
$ cat /var/log/syslog|head -n 4 Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Starting Flush Journal to Persistent Storage... Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Set the console keyboard layout. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load/Save Random Seed. Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
$ cat /var/log/syslog|tail -n 4 Feb 25 20:13:38 overfittedropout-virtual-machine snapd[806]: storehelpers.go:769: cannot refresh: snap has no updates available: "bare", "core20", "core22", "firefox", "gnome-3-38-2004", "gnome-42-2204", "gtk-common-themes", "snap-store", "snapd", "snapd-desktop-integration" Feb 25 20:13:38 overfittedropout-virtual-machine snapd[806]: autorefresh.go:551: auto-refresh: all snaps are up-to-date Feb 25 20:14:05 overfittedropout-virtual-machine systemd[1]: systemd-timedated.service: Deactivated successfully. Feb 25 20:17:01 overfittedropout-virtual-machine CRON[2213]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
In the above example, we printed the first and last 4 lines of the file. You can view additional options and syntax using "man tail" and "man head".
For viewing logs which are continuously appended in real-time (e.g. catalina.out), we can view the latest content by adding "-f" flag to tail command:
tail -2f /var/log/syslog Mar 19 13:06:18 overfittedropout-virtual-machine NetworkManager[853]: [1679211378.7842] manager: NetworkManager state is now CONNECTED_GLOBAL Mar 19 13:06:28 overfittedropout-virtual-machine systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully. Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Activating service name='org.gnome.Nautilus' requested by ':1.24' (uid=1000 pid=1292 comm="/usr/bin/gnome-shell" label="unconfined") Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Successfully activated service 'org.gnome.Nautilus'
The above command starts printing logs from the last two lines of the file and prints further as the file is appended with new information.
Enabling/Disabling system logs through rsyslog
When you start exploring log files, you might see that there are several log files which might not be present under "/var/log/". You can enable the log files in the rsyslog configuration file.
a. Navigate to "/etc/rsyslog.d".
$ cd /etc/rsyslog.d $ ls -ltr total 12 -rw-r--r-- 1 root root 314 Sep 19 2021 20-ufw.conf -rw-r--r-- 1 root root 1121 Feb 23 22:41 50-default.conf.save -rw-r--r-- 1 root root 1114 Feb 23 23:18 50-default.con
b. In the above screenshot, we can see the configuration files. We will be working with '50-default.conf'
c. Please ensure to open the file in root user mode to edit it.
sudo nano 50-default.conf # Default rules for rsyslog. # # For more information see rsyslog.conf(5) and /etc/rsysl> # # First some standard log files. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog cron.* /var/log/cron.log #daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log #lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log
In the above excerpt, we can see the daemon logs are commented out with '#'. To enable daemon logs, we can remove the # and save the file. We can also see the path where the logs will be stored and this can be changed as well.
Please ensure to restart the system for the configurations to reflect. Similarly, we can also disable any logs by commenting it out with '#'.
Further readings
Logs are a very important component for any Linux user. After mastering the basics, you can go ahead and explore more about log analysis.
In depth explanation of log generation – Linux Logging Complete Guide – devconnected
Linux logs + SIEM solution – Monitoring Linux Server with Splunk - Patrick Bareiss (patrick-bareiss.com)
