373 lines
13 KiB
Markdown
373 lines
13 KiB
Markdown
|
|
Tmux sysstat plugin
|
|
===================
|
|
Allow to print CPU usage, memory & swap, load average, net I/O metrics in Tmux status bar
|
|
|
|
|
|
|
|
You might checkout [tmux-config](https://github.com/samoshkin/tmux-config) repo to see this plugin in action.
|
|
|
|
Features
|
|
--------
|
|
- CPU usage
|
|
- Memory available/free, used, total (KiB,MiB,GiB), free/used %
|
|
- Swap used, free, total, free/used %
|
|
- load average for last 1,5,15 minutes
|
|
- configurable thresholds (low, medium, stress) with custom colors
|
|
- tweak each metric output using templates (e.g, 'used 10% out of 16G')
|
|
- configurable size scale (K,M,G)
|
|
- OSX, Linux support
|
|
- [ ] **TODO:** network I/O metric support
|
|
|
|
Tested on: OS X El Capitan 10.11.5, Ubuntu 14 LTS, CentOS 7, FreeBSD 11.1.
|
|
|
|
|
|
Installation
|
|
------------
|
|
Best installed through [Tmux Plugin Manager](https://github.com/tmux-plugins/tpm) (TMP). Add following line to your `.tmux.conf` file:
|
|
|
|
```
|
|
set -g @plugin 'samoshkin/tmux-plugin-sysstat'
|
|
```
|
|
|
|
Use `prefix + I` from inside tmux to install all plugins and source them. If you prefer, same effect can be achieved from [command line](https://github.com/tmux-plugins/tpm/blob/master/docs/managing_plugins_via_cmd_line.md):
|
|
|
|
```
|
|
$ ~.tmux/plugins/tpm/bin/install_plugins
|
|
```
|
|
|
|
Basic usage
|
|
-----------
|
|
|
|
Once plugged in, tmux `status-left` or `status-right` options can be configured with following placeholders. Each placeholder will be expanded to metric's default output.
|
|
|
|
- `#{sysstat_cpu}`, CPU usage - `CPU:40.2%`
|
|
- `#{sysstat_mem}`, memory usage - `MEM:73%`
|
|
- `#{sysstat_swap}`, swap usage - `SW:66%`
|
|
- `#{sysstat_loadavg}`, system load average - `0.25 0.04 0.34`
|
|
|
|
For example:
|
|
```
|
|
set -g status-right "#{sysstat_cpu} | #{sysstat_mem} | #{sysstat_swap} | #{sysstat_loadavg} | #[fg=cyan]#(echo $USER)#[default]@#H"
|
|
```
|
|
|
|
Changing default output
|
|
------------------------
|
|
|
|
You can change default output for CPU and memory metrics, if you need more fields to show, or you want to provide custom template. In your `.tmux.conf`:
|
|
|
|
For example, to get `Used 4.5G out of 16G` output for memory metric:
|
|
|
|
```
|
|
set -g @sysstat_mem_view_tmpl '#Used [fg=#{mem.color}]#{mem.used}#[default] out of #{mem.total}'
|
|
```
|
|
|
|
If you don't want `CPU:` prefix and don't like colored output for CPU metric:
|
|
|
|
```
|
|
set -g @sysstat_cpu_view_tmpl '#{cpu.pused}'
|
|
```
|
|
|
|
### Supported fields
|
|
|
|
As you can see, each metric can be configured with template, containing fixed text (`CPU:`), color placeholder (`#[fg=#{mem.color}]`) and field placeholder (`#{mem.used}`). This approach gives you the ultimate control over the output for each metric. Following field placeholders are supported:
|
|
|
|
<table>
|
|
<th>CPU</th>
|
|
<tr>
|
|
<td><code>#{cpu.color}</code></td>
|
|
<td>main metric color</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{cpu.pused}</code></td>
|
|
<td>CPU usage percentage</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<th>Memory</th>
|
|
<tr>
|
|
<td><code>#{mem.color}</code></td>
|
|
<td>main metric color</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{mem.free}</code></td>
|
|
<td>free/available memory</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{mem.pfree}</code></td>
|
|
<td>free memory percentage against total</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{mem.used}</code></td>
|
|
<td>used memory</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{mem.pused}</code></td>
|
|
<td>used memory percentage against total</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{mem.total}</code></td>
|
|
<td>total installed memory</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<th>Swap</th>
|
|
<tr>
|
|
<td><code>#{swap.color}</code></td>
|
|
<td>main swap metric color</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{swap.free}</code></td>
|
|
<td>free swap memory</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{swap.pfree}</code></td>
|
|
<td>free swap memory percentage against total swap space</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{swap.used}</code></td>
|
|
<td>used swap memory</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{swap.pused}</code></td>
|
|
<td>used swap memory percentage against total swap space</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>#{swap.total}</code></td>
|
|
<td>total swap space</td>
|
|
</tr>
|
|
</table>
|
|
|
|
### Change size scale
|
|
|
|
free/used/total memory can be shown both in absolute and relative units. When it comes to absolute units, you can choose *size scale factor* to choose between GiB, MiB, KiB. Default is GiB. If you have less than 3-4G memory installed, it makes sense to use MiB. KiB option is less practical, because it yields pretty lengthy output, which does not fit status bar limited estate.
|
|
|
|
```
|
|
set -g @sysstat_mem_size_unit "G"
|
|
```
|
|
|
|
If you choose `G` for size scale, output will have `%.1f` (1 digit after floating point), otherwise size is integer (4.5G, 1024M, 1232345K).
|
|
|
|
Thresholds and colored output
|
|
---------------
|
|
Each metric output is colored by default. Colors vary depending on metric value.
|
|
|
|
<table>
|
|
<tr>
|
|
<td><b>Threshold</b></td>
|
|
<td><b>CPU</b></td>
|
|
<td><b>Memory</b></td>
|
|
<td><b>Swap</b></td>
|
|
<td><b>Default color</b></td>
|
|
</tr>
|
|
<tr>
|
|
<td>low</td>
|
|
<td>x < 30%</td>
|
|
<td>x < 75%</td>
|
|
<td>x < 25%</td>
|
|
<td>green</td>
|
|
</tr>
|
|
<tr>
|
|
<td>medium</td>
|
|
<td>30% < x < 80%</td>
|
|
<td>75% < x < 90%</td>
|
|
<td>25% < x < 75%</td>
|
|
<td>yellow</td>
|
|
</tr>
|
|
<tr>
|
|
<td>high</td>
|
|
<td>x > 80%</td>
|
|
<td>x > 90%</td>
|
|
<td>x > 75%</td>
|
|
<td>red</td>
|
|
</tr>
|
|
</table>
|
|
|
|
You can change thresholds in your `.tmux.conf`:
|
|
|
|
```
|
|
set -g @sysstat_cpu_medium_threshold "75"
|
|
set -g @sysstat_cpu_stress_threshold "95"
|
|
|
|
set -g @sysstat_mem_medium_threshold "85"
|
|
set -g @sysstat_mem_stress_threshold "95"
|
|
|
|
set -g @sysstat_swap_medium_threshold "80"
|
|
set -g @sysstat_swap_stress_threshold "90"
|
|
```
|
|
|
|
You can change colors for each threshold individually. You can use ANSI basic colors (red, cyan, green) or if your terminal supports 256 colors (and most do nowadays), use `colourXXX` format.
|
|
|
|
```
|
|
set -g @sysstat_cpu_color_low "colour076"
|
|
set -g @sysstat_cpu_color_medium "colour220"
|
|
set -g @sysstat_cpu_color_stress "colour160"
|
|
set -g @sysstat_mem_color_low "green"
|
|
set -g @sysstat_mem_color_medium "blue"
|
|
set -g @sysstat_mem_color_stress "cyan"
|
|
```
|
|
|
|
`#{(mem|cpu|swap).color}` placeholder in your `@sysstat_(mem|cpu|swap)_view_tmpl` would be replaced by corresponding color, depending on whether metric value falls in particular threshold.
|
|
|
|
### 256 color palette support
|
|
|
|
For 256 color palette support, make sure that `tmux` and parent terminal are configured with correct terminal type. See [here](https://unix.stackexchange.com/questions/1045/getting-256-colors-to-work-in-tmux) and [there](https://github.com/tmux/tmux/wiki/FAQ)
|
|
|
|
```
|
|
# ~/.tmux.conf
|
|
set -g default-terminal "screen-256color"
|
|
```
|
|
|
|
```
|
|
# parent terminal
|
|
$ echo $TERM
|
|
xterm-256color
|
|
|
|
# jump into a tmux session
|
|
$ tmux new
|
|
$ echo $TERM
|
|
screen-256color
|
|
```
|
|
|
|
|
|
|
|
### Multiple colors for each threshold
|
|
|
|
You can have up to *3* colors configured for each threshold. To understand why you might need this, let tackle this task. Note, this is rather advanced use case.
|
|
|
|
> I want `CPU: #{cpu.pused}` metric output, have green and yellow text colors at "low" and "medium" threshold, and finally, for "high" threshold, I want to use red color, but reverse foreground and background, that is use red for background, and white for text. More over I want "CPU:" text colored apart in red
|
|
|
|
Like this:
|
|
|
|
|
|
|
|
You can achieve the result using following configuration:
|
|
|
|
```
|
|
set -g @sysstat_cpu_view_tmpl '#[fg=#{cpu.color3}]CPU:#[default] #[fg=#{cpu.color},bg=#{cpu.color2}]#{cpu.pused}#[default]'
|
|
|
|
set -g @sysstat_cpu_color_low "$color_level_ok default default"
|
|
set -g @sysstat_cpu_color_medium "$color_level_warn default default"
|
|
set -g @sysstat_cpu_color_stress "white,bold $color_level_stress $color_level_stress"
|
|
```
|
|
|
|
Tmux status-interval setting
|
|
-----------------------------
|
|
You can configure status refresh interval, increasing or reducing frequency of `tmux-plugin-sysstat` command invocations.
|
|
|
|
```
|
|
set -g status-interval 5
|
|
```
|
|
|
|
It's adviced to set `status-interval` to some reasonable value, like 5-10 seconds. More frequent updates (1 second) are useless, because they distract, and results in extra resource stress spent on metrics calculation itself.
|
|
|
|
|
|
|
|
Internals: CPU calculation
|
|
--------------------------------------------------
|
|
<span style="color: blue">**NOTE:** Stop here if you want to just use this plugin without making your feet wet. If you're hardcore tmux user and are curious about internals, keep reading</span>
|
|
|
|
Internally, we use `iostat` and `top` on OSX, and `vmstat` and `top` on Linux to collect metric value. Neither requires you to install extra packages. These commands are run in sampling mode to report stats every N seconds M times. First sample include average values since the system start. Second one is the average CPU per second for last N seconds (exactly what we need)
|
|
|
|
For example:
|
|
|
|
```
|
|
$ iostat -c 2 -w 5
|
|
disk0 cpu load average
|
|
KB/t tps MB/s us sy id 1m 5m 15m
|
|
44.22 6 0.26 3 2 95 1.74 1.90 2.15
|
|
5.47 8 0.04 4 5 91 1.84 1.92 2.16 << use this row, 2nd sample
|
|
```
|
|
|
|
We align CPU calculation intervals (`-w`) with tmux status bar refresh interval (`status-interval` setting).
|
|
|
|
Internals: memory calculation
|
|
----------------------------
|
|
You might ask what we treat as `free` memory and how it's calculated.
|
|
|
|
### OSX
|
|
Let's start with OSX. We use `vm_stat` command (not same as `vmstat` on Linux), which reports following data (number of memory pages, not KB):
|
|
|
|
```
|
|
$ vm_stat
|
|
Pages free: 37279
|
|
Pages active: 1514200
|
|
Pages inactive: 1152997
|
|
Pages speculative: 6214
|
|
Pages throttled: 0
|
|
Pages wired down: 1174408
|
|
Pages purgeable: 15405
|
|
Pages stored in compressor: 1615663
|
|
Pages occupied by compressor: 306717
|
|
```
|
|
|
|
Total installed memory formula is:
|
|
```
|
|
Total = free + active + inactive + speculative + occupied by compressor + wired
|
|
```
|
|
|
|
where
|
|
- `free`, completely unused memory by the system
|
|
- `wired`, critical information stored in RAM by system, kernel and key applications. Never swapped to the hard drive, never replaced with user-level data.
|
|
- `active`, information currently in use or very recently used by applications. When this kind of memory is not used for long (or application is closed), it's move to inactive memory.
|
|
- `inactive`, like buffers/cached memory in Linux. Memory for applications, which recently exited, retained for faster start-up of same application in future.
|
|
|
|
So the question what constitutes `free` and `used` memory. It turns out, that various monitoring and system statistics tools on OSX each calculate it differently.
|
|
|
|
- htop: `used = active + wired`, `free` = `total - used`
|
|
- top. Used = `used = active + inactive + occupied by compressor + wired`; Free = `free + speculative` Resident set size (RSS) = `active`
|
|
- OSX activity Monitor. Used = `app memory + wired + compressor`. Note, it's not clear what is app memory.
|
|
|
|
In general, they either treat currently used memory, which can be reclaimed in case of need (cached, inactive, occupied by compressor), as `used` or `free`.
|
|
|
|
It makes sense to talk about `available` memory rather than `free` one. Available memory is unused memory + any used memory which can be reclaimed for application needs.
|
|
|
|
So, `tmux-plugin-sysstat`, uses following formula:
|
|
|
|
```
|
|
used = active + wired
|
|
available/free = free/unused + inactive + speculative + occupied by compressor
|
|
```
|
|
|
|
### Linux
|
|
|
|
Same thinking can be applied to Linux systems.
|
|
|
|
Usually commands like `free` report free/unused, used, buffers, cache memory kinds.
|
|
|
|
```
|
|
$ free
|
|
total used free shared buffers cached
|
|
Mem: 1016464 900236 116228 21048 93448 241544
|
|
-/+ buffers/cache: 565244 451220
|
|
Swap: 1046524 141712 904812
|
|
```
|
|
|
|
Second line indicates available memory (free + buffers + cache), with an assumption that buffers and cache can be 100% reclaimed in case of need.
|
|
|
|
However, we're not using free, because its output varies per system. For example on RHEL7, there is no `-/+ buffers/cache`, and `available` memory is reported in different way. We read directly from `/proc/meminfo`
|
|
|
|
```
|
|
$ cat /proc/meminfo
|
|
|
|
MemTotal: 1016232 kB
|
|
MemFree: 152672 kB
|
|
MemAvailable: 637832 kB
|
|
Buffers: 0 kB
|
|
Cached: 529040 kB
|
|
```
|
|
|
|
`tmux-plugin-sysstat` uses following formula:
|
|
|
|
```
|
|
free/available = MemAvailable; // if MemAvailable present
|
|
free/available = MemFree + Buffers + Cached;
|
|
used = MemTotal - free/avaialble
|
|
```
|
|
|
|
Using `MemAvailable` is more accurate way of getting available memory, rather than manual calculation `free + buffers + cache`, because the assumption that `buffers + cache` can be 100% reclaimed for new application needs might be wrong. When using `MemAvailable`, OS calculates available memory for you, which is apparently better and accurate approach.
|
|
|
|
See [this topic](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773) on more reasoning about `MemAvailable` field.
|