Tool and tips on building & configuring collectd (on Ubuntu)
collectd is default standard mechanism to collect systems statistics from ICT infrastructure. However building it from source code is non-trivial due to the large number of plugins. Lessons learnt, tip and tools to make this easier.
Welcome to my collectd build and configuration tips. Telemetry is an essential element of operations automation and collectd (along with fluentd) are the tools to instrument your infrastructure.
I am not going to talk about merits of collectd vs. fluentd here. Simply put if you are using OpenStack / QEMU / libvirt then collectd is your friend and if you are using Kubernetes then fluentd is your friend. In a large cloud deployment expect to be friends with both of these and buddy them up with Prometheus and Kafka to get your real time streaming telemetry solution going.
Upstream collectd
collectd do is now a very mature set code with an active community and a very large set of plug-ins to extract: metrics, events and information from a large number of of components, across physical and virtual infrastructure and sofware componets like web servers, databases, storage and all else and sundry.
With so many plug-ins currency will always be an ongoing concern and this was what got me into collectd contribution. I was building KVM host on Ubuntu 19.04 (as part of telemetry technical proof) and this used updated libvirt version which resulting in libvirt / virt plug-in currency bug. To get this resolved I took a fork and then hit another tooling currency issue. As often happens I ended up wading in much deeper than expected. In doing this I learnt a lot about the collectd build system and created a number of little scripts to help with doing upstream collectd builds. I am documenting these and making them available to make it easier for to contribute to collectd development.
Build Tool Chain
The collectd build chain can be very very long indeed. The reason is that collectd automatic configuration process detects which of the various libraries are installed and then based on this configures you build process appropriately. This means that you can contain the scope of your build by pulling in or removing various dependencies.
The core build chain requires: autoconf, automake, bison, build-essential, clang-format, git, flex, libtool:
autoconf automake bison build-essential clang-format git flex libtool
Package Dependencies...
Getting the build tools is the simple part... if you want to build a full set of collectd plug-ins for regression build test then you will need to install a lot of packages. It took me quite a bit of seaching to find the right set. Here is my current list for Ubuntu 19.04:
intel-cmt-cat iptables-dev libcephfs-dev libcurl4-gnutls-dev libfreeipmi-dev libganglia1-dev libgps-dev libgrpc++ libgrpc++-dev libgrpc-dev libldap2-dev liblua5.1-0-dev liblvm2app2.2 liblvm2-dev libmemcached-dev libmicrohttpd-dev libmysqlclient-dev libndpi-dev libnetapp-perl libow-dev libperl-dev libprotobuf17 libprotobuf-c-dev libprotobuf-dev libprotoc-dev librabbitmq-dev librdkafka-dev libriemann-client-dev librrd8 libsensors4-dev libsigrok-dev libsnmp-dev libvarnishapi-dev libvirt-dev libxml2 libxml2-dev mcollective-plugins-logstash mongodb-dev mysql-client perl pforth postgresql-client protobuf-c-compiler python-protobuf redis rrdtool ruby-google-protobuf sigrok
To make this simpler I created a small script. This takes a candidate set of installation packages (and optional "test" flag). To help discover if you have the right installation package it does "apt search" first before doing "apt install". This lets you run test against result from ./configure to find packages required to build the modules.
An example of this is configure results (see below) indicates "amqp" module but the Ubuntu apt package required is "librabbitmq-dev" do the "apt search" helps with find the appropriate dependency packages. Simply took the "Libraries" result list (after feed to through sed to clean up input) and feed the names into the get-modules.sh script below to help get the right set of packages.
#!/bin/sh APT="apt" TEST=0 if [ $# -eq 0 ]; then echo "Usage: $0[test]" exit 1 fi if [ $2 = test ]; then TEST=1 fi #LIB-LIST="`cat $1`" res=0 for lib in `cat $1` ; do (apt search ${lib} 2> /dev/null | wc | grep -q " 2 ") if [ $? -eq 0 ]; then echo "not found: ${lib}" elif [ ${TEST} -eq 1 ]; then echo "would do: sudo apt install ${lib}" else echo "doing: sudo apt install ${lib}" sudo apt install ${lib} fi done
Example ./configure result look like this (with Libraries & Modules), which need to be mapped to Ubuntu / Debian installation package:
Libraries:
intel mic . . . . . . no (MicAccessApi not found)
libaquaero5 . . . . . no (libaquaero5.h not found)
libatasmart . . . . . yes
libcurl . . . . . . . yes
libdbi . . . . . . . yes
libdpdk . . . . . . . yes
libesmtp . . . . . . yes
libganglia . . . . . yes
libgcrypt . . . . . . yes
libgps . . . . . . . yes
libgrpc++ . . . . . . yes
libhiredis . . . . . yes
libi2c-dev . . . . . yes
libiokit . . . . . . no
libiptc . . . . . . . yes
libjevents . . . . . no (jevents.h not found)
libjvm . . . . . . . no (jar not found)
libkstat . . . . . . no (Solaris only)
libkvm . . . . . . . no
libldap . . . . . . . yes
liblua . . . . . . . yes
liblvm2app . . . . . yes
libmemcached . . . . yes
libmicrohttpd . . . . yes
libmnl . . . . . . . yes
libmodbus . . . . . . yes
libmongoc . . . . . . yes
libmosquitto . . . . yes
libmysql . . . . . . yes
libnetapp . . . . . . no (netapp_api.h not found)
libnetsnmp . . . . . yes
libnetsnmpagent . . . yes
libnotify . . . . . . yes
libnvidia-ml . . . . no
libopenipmi . . . . . yes
liboping . . . . . . yes
libowcapi . . . . . . yes
libpcap . . . . . . . yes
libperfstat . . . . . no (AIX only)
libperl . . . . . . . yes (version 5.28.1)
libpq . . . . . . . . yes
libpqos . . . . . . . yes
libprotobuf . . . . . yes
libprotobuf-c . . . . yes
libpython . . . . . . no
libqpid-proton . . . yes
librabbitmq . . . . . yes
libriemann-client . . yes
librdkafka . . . . . yes
librouteros . . . . . no (routeros_api.h not found)
librrd . . . . . . . yes
libsensors . . . . . yes
libsigrok . . . . . no (pkg-config could not find libsigrok)
libssl . . . . . . . yes
libstatgrab . . . . . no (pkg-config doesn't know libstatgrab)
libtokyotyrant . . . yes
libudev . . . . . . . yes
libupsclient . . . . yes
libvarnish . . . . . yes
libvirt . . . . . . . yes
libxenctrl . . . . . yes
libxml2 . . . . . . . yes
libxmms . . . . . . . no
libyajl . . . . . . . yes
oracle . . . . . . . no (ORACLE_HOME is not set)
protobuf-c . . . . . yes
protoc 3 . . . . . . no
Features:
daemon mode . . . . . yes
debug . . . . . . . . no
Bindings:
perl . . . . . . . . yes (PREFIX=NONE)
Modules:
aggregation . . . . . yes
amqp . . . . . . . yes
amqp1 . . . . . . . yes
apache . . . . . . . yes
apcups . . . . . . . yes
apple_sensors . . . . no
aquaero . . . . . . . no (libaquaero5.h not found)
ascent . . . . . . . yes
barometer . . . . . . yes
battery . . . . . . . yes
bind . . . . . . . . yes
ceph . . . . . . . . yes
cgroups . . . . . . . yes
chrony. . . . . . . . yes
conntrack . . . . . . yes
contextswitch . . . . yes
cpu . . . . . . . . . yes
cpufreq . . . . . . . yes
cpusleep . . . . . . yes
csv . . . . . . . . . yes
curl . . . . . . . . yes
curl_json . . . . . . yes
curl_xml . . . . . . yes
dbi . . . . . . . . . yes
df . . . . . . . . . yes
disk . . . . . . . . yes
dns . . . . . . . . . yes
dpdkevents. . . . . . yes
dpdkstat . . . . . . yes
drbd . . . . . . . . yes
email . . . . . . . . yes
entropy . . . . . . . yes
ethstat . . . . . . . yes
exec . . . . . . . . yes
fhcount . . . . . . . yes
filecount . . . . . . yes
fscache . . . . . . . yes
gmond . . . . . . . . yes
gps . . . . . . . . . yes
gpu_nvidia . . . . . no
grpc . . . . . . . . no (protoc3 not found)
hddtemp . . . . . . . yes
hugepages . . . . . . yes
intel_pmu . . . . . . no (jevents.h not found)
intel_rdt . . . . . . yes
interface . . . . . . yes
ipc . . . . . . . . . yes
ipmi . . . . . . . . yes
iptables . . . . . . yes
ipvs . . . . . . . . yes
irq . . . . . . . . . yes
java . . . . . . . . no (jar not found)
load . . . . . . . . yes
logfile . . . . . . . yes
log_logstash . . . . yes
lpar . . . . . . . . no (AIX only)
lua . . . . . . . . . yes
lvm . . . . . . . . . yes
madwifi . . . . . . . yes
match_empty_counter . yes
match_hashed . . . . yes
match_regex . . . . . yes
match_timediff . . . yes
match_value . . . . . yes
mbmon . . . . . . . . yes
mcelog . . . . . . . yes
md . . . . . . . . . yes
memcachec . . . . . . yes
memcached . . . . . . yes
memory . . . . . . . yes
mic . . . . . . . . . no (MicAccessApi not found)
modbus . . . . . . . yes
mqtt . . . . . . . . yes
multimeter . . . . . yes
mysql . . . . . . . . yes
netapp . . . . . . . no (netapp_api.h not found)
netlink . . . . . . . yes
network . . . . . . . yes
nfs . . . . . . . . . yes
nginx . . . . . . . . yes
notify_desktop . . . yes
notify_email . . . . yes
notify_nagios . . . . yes
ntpd . . . . . . . . yes
numa . . . . . . . . yes
nut . . . . . . . . . yes
olsrd . . . . . . . . yes
onewire . . . . . . . yes
openldap . . . . . . yes
openvpn . . . . . . . yes
oracle . . . . . . . no (ORACLE_HOME is not set)
ovs_events . . . . . yes
ovs_stats . . . . . . yes
pcie_errors . . . . . yes
perl . . . . . . . . yes
pf . . . . . . . . . no
pinba . . . . . . . . yes
ping . . . . . . . . yes
postgresql . . . . . yes
powerdns . . . . . . yes
processes . . . . . . yes
protocols . . . . . . yes
python . . . . . . . no
redis . . . . . . . . yes
routeros . . . . . . no (routeros_api.h not found)
rrdcached . . . . . . yes
rrdtool . . . . . . . yes
sensors . . . . . . . yes
serial . . . . . . . yes
sigrok . . . . . . . no (pkg-config could not find libsigrok)
smart . . . . . . . . yes
snmp . . . . . . . . yes
snmp_agent . . . . . yes
statsd . . . . . . . yes
swap . . . . . . . . yes
synproxy . . . . . . yes
syslog . . . . . . . yes
table . . . . . . . . yes
tail_csv . . . . . . yes
tail . . . . . . . . yes
tape . . . . . . . . no
target_notification . yes
target_replace . . . yes
target_scale . . . . yes
target_set . . . . . yes
target_v5upgrade . . yes
tcpconns . . . . . . yes
teamspeak2 . . . . . yes
ted . . . . . . . . . yes
thermal . . . . . . . yes
threshold . . . . . . yes
tokyotyrant . . . . . yes
turbostat . . . . . . yes
unixsock . . . . . . yes
uptime . . . . . . . yes
users . . . . . . . . yes
uuid . . . . . . . . yes
varnish . . . . . . . yes
virt . . . . . . . . yes
vmem . . . . . . . . yes
vserver . . . . . . . yes
wireless . . . . . . yes
write_graphite . . . yes
write_http . . . . . yes
write_kafka . . . . . yes
write_log . . . . . . yes
write_mongodb . . . . yes
write_prometheus. . . yes
write_redis . . . . . yes
write_riemann . . . . yes
write_sensu . . . . . yes
write_stackdriver . . yes
write_syslog . . . . yes
write_tsdb . . . . . yes
xencpu . . . . . . . yes
xmms . . . . . . . . no
zfs_arc . . . . . . . yes
zone . . . . . . . . no
zookeeper . . . . . . yes
Closing Throughts: These notes / helper tools are obviously specific to Ubuntu / Debian, but I believe that it would be useful to have simillar set of build asssistance tools for other platforms (RHEL, FreeBSD & Solaris as each of these has some OS specific plug-ins), as building upsteam collectd is not simple. I have suggested this within collectd pull request comments.
My First collectd Pull Request: https://github.com/collectd/collectd/pull/3153 a small step for open source, but a big step for an "enterprise architect" ;-)