How to monitor Nvidia GPU temperature on Ubuntu / Linux

GPU temperature

Nvidia graphics card has its own processor that gets heated just like the main cpu processor. They usually have a heat sink and the high power gpus also have a fan.

Its a good idea to monitor the gpu temperature on your system from time to time to ensure that the unit is not overheating and keeping good.

On Ubuntu, you need to first install the Nvidia drivers. That will install additional utility programs that can monitor and report various statistics about the gpu.

nvidia-settings

The "Nvidia X Server Settings" is a gui program that reports details about the gpu. You can find the tool in the menu, after installing the nvidia drivers. To launch from the command line, run the command nvidia-settings

Here is a screenshot.

The nvidia-settings command can also be used to monitor the gpu usage.

$ nvidia-settings -q GPUUtilization

  Attribute 'GPUUtilization' (desktop:0[gpu:0]): graphics=27, memory=20, video=0, PCIe=0

The graphics=27 is gpu utilization level. To monitor continuously use the watch command

$ watch -n 1 nvidia-settings -q GPUUtilization

nvidia-smi

The nvidia-smi command shows the temperature of the gpu. Go to GPU > Thermal settings to check the temperature.

$ nvidia-smi
Sun Feb 19 18:44:33 2017       
+------------------------------------------------------+                       
| NVIDIA-SMI 340.102    Driver Version: 340.102        |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 210         Off  | 0000:01:00.0     N/A |                  N/A |
| N/A   62C   P12    N/A /  N/A |    233MiB /  1023MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0            Not Supported                                               |
+-----------------------------------------------------------------------------+

Check the 64C figure on the middle left box. The middle box shows the memory usage.
You can monitor it continuously with watch command -

watch -n 1 nvidia-smi

Check what processes are using the gpu

The following command can be used to check, what processes are accessing the gpu -

ps f -o user,pgrp,pid,pcpu,pmem,start,time,command -p `lsof -n -w -t /dev/nvidia*`

Source

Thermal Monitor KDE Plasmoid

The Kde plasmoid named "Thermal Monitor" can be used to monitor sensors via lm-sensor. First you need to install lm-sensors package

sudo apt-get install lm-sensors

Then install the KDE plasmoid from "Get New Widgets" dialog. Right click on the desktop and click Add new widget and then Get new widgets.

Place the widget on the desktop and configure the sensors to display.

GPU Overheating

Most low price gpus do not have a cooling fan and rely only on the heat sink. However, they do get very hot depending on what applications are using it.

If your nvidia gpu is showing signs of over heating, then consider adding an extra casing fan to the pc case near the gpu unit to cool it down. A fan blowing air on the gpu can cool it over 15 C.