Tool and tips on building & configuring collectd (on Ubuntu)

Welcome to my collectd build and configuration tips. Telemetry is an essential element of operations automation and collectd (along with fluentd) are the tools to instrument your infrastructure.

I am not going to talk about merits of collectd vs. fluentd here. Simply put if you are using OpenStack / QEMU / libvirt then collectd is your friend and if you are using Kubernetes then fluentd is your friend. In a large cloud deployment expect to be friends with both of these and buddy them up with Prometheus and Kafka to get your real time streaming telemetry solution going.

Upstream collectd

collectd do is now a very mature set code with an active community and a very large set of plug-ins to extract: metrics, events and information from a large number of of components, across physical and virtual infrastructure and sofware componets like web servers, databases, storage and all else and sundry.

With so many plug-ins currency will always be an ongoing concern and this was what got me into collectd contribution. I was building KVM host on Ubuntu 19.04  (as part of telemetry technical proof) and this used updated libvirt version which resulting in libvirt / virt plug-in currency bug.  To get this resolved I took a fork and then hit another tooling currency issue. As often happens I ended up wading in much deeper than expected. In doing this I learnt a lot about the collectd build system and created a number of little scripts to help with doing upstream collectd builds. I am documenting these and making them available to make it easier for to contribute to collectd development.

Build Tool Chain

The collectd build chain can be very very long indeed. The reason is that collectd automatic configuration process detects which of the various libraries are installed and then based on this configures you build process appropriately. This means that you can contain the scope of your build by pulling in or removing various dependencies.

The core build chain requires: autoconf, automake, bison, build-essential, clang-format, git, flex, libtool:

autoconf
automake
bison
build-essential
clang-format
git
flex
libtool

Package Dependencies...

Getting the build tools is the simple part... if you want to build a full set of collectd plug-ins for regression build test then you will need to install a lot of packages. It took me quite a bit of seaching to find the right set. Here is my current list for Ubuntu 19.04:

intel-cmt-cat
iptables-dev
libcephfs-dev
libcurl4-gnutls-dev
libfreeipmi-dev
libganglia1-dev
libgps-dev
libgrpc++
libgrpc++-dev
libgrpc-dev
libldap2-dev
liblua5.1-0-dev
liblvm2app2.2
liblvm2-dev
libmemcached-dev
libmicrohttpd-dev
libmysqlclient-dev
libndpi-dev
libnetapp-perl
libow-dev
libperl-dev
libprotobuf17
libprotobuf-c-dev
libprotobuf-dev
libprotoc-dev
librabbitmq-dev
librdkafka-dev
libriemann-client-dev
librrd8
libsensors4-dev
libsigrok-dev
libsnmp-dev
libvarnishapi-dev
libvirt-dev
libxml2
libxml2-dev
mcollective-plugins-logstash
mongodb-dev
mysql-client
perl
pforth
postgresql-client
protobuf-c-compiler
python-protobuf
redis
rrdtool
ruby-google-protobuf
sigrok

To make this simpler I created a small script. This takes a candidate set of installation packages (and optional "test" flag). To help discover if you have the right installation package it does "apt search" first before doing "apt install". This lets you run test against result from ./configure to find packages required to build the modules.

An example of this is configure results (see below) indicates "amqp" module but the Ubuntu apt package required is "librabbitmq-dev" do the "apt search" helps with find the appropriate dependency packages. Simply took the "Libraries" result list (after feed to through sed to clean up input) and feed the names into the get-modules.sh script below to help get the right set of packages.

#!/bin/sh

APT="apt"
TEST=0

if [ $# -eq 0 ];
then
  echo "Usage: $0  [test]"
  exit 1
fi

if [ $2 = test ];
then
  TEST=1
fi

#LIB-LIST="`cat $1`"
res=0

for lib in `cat $1` ;
do
  (apt search ${lib} 2> /dev/null | wc | grep -q " 2 ")
  if [ $? -eq 0 ];
  then
    echo "not found: ${lib}"
  elif [ ${TEST} -eq 1 ];
  then
    echo "would do: sudo apt install ${lib}"
  else
    echo "doing: sudo apt install ${lib}"
    sudo apt install ${lib}
  fi
done

Example ./configure result look like this (with Libraries & Modules), which need to be mapped to Ubuntu / Debian installation package:

  Libraries:
    intel mic . . . . . . no (MicAccessApi not found)
    libaquaero5 . . . . . no (libaquaero5.h not found)
    libatasmart . . . . . yes
    libcurl . . . . . . . yes
    libdbi  . . . . . . . yes
    libdpdk . . . . . . . yes
    libesmtp  . . . . . . yes
    libganglia  . . . . . yes
    libgcrypt . . . . . . yes
    libgps  . . . . . . . yes
    libgrpc++ . . . . . . yes
    libhiredis  . . . . . yes
    libi2c-dev  . . . . . yes
    libiokit  . . . . . . no
    libiptc . . . . . . . yes
    libjevents  . . . . . no (jevents.h not found)
    libjvm  . . . . . . . no (jar not found)
    libkstat  . . . . . . no (Solaris only)
    libkvm  . . . . . . . no
    libldap . . . . . . . yes
    liblua  . . . . . . . yes
    liblvm2app  . . . . . yes
    libmemcached  . . . . yes
    libmicrohttpd . . . . yes
    libmnl  . . . . . . . yes
    libmodbus . . . . . . yes
    libmongoc . . . . . . yes
    libmosquitto  . . . . yes
    libmysql  . . . . . . yes
    libnetapp . . . . . . no (netapp_api.h not found)
    libnetsnmp  . . . . . yes
    libnetsnmpagent . . . yes
    libnotify . . . . . . yes
    libnvidia-ml  . . . . no
    libopenipmi . . . . . yes
    liboping  . . . . . . yes
    libowcapi . . . . . . yes
    libpcap . . . . . . . yes
    libperfstat . . . . . no (AIX only)
    libperl . . . . . . . yes (version 5.28.1)
    libpq . . . . . . . . yes
    libpqos . . . . . . . yes
    libprotobuf . . . . . yes
    libprotobuf-c . . . . yes
    libpython . . . . . . no
    libqpid-proton .  . . yes
    librabbitmq . . . . . yes
    libriemann-client . . yes
    librdkafka  . . . . . yes
    librouteros . . . . . no (routeros_api.h not found)
    librrd  . . . . . . . yes
    libsensors  . . . . . yes
    libsigrok   . . . . . no (pkg-config could not find libsigrok)
    libssl  . . . . . . . yes
    libstatgrab . . . . . no (pkg-config doesn't know libstatgrab)
    libtokyotyrant  . . . yes
    libudev . . . . . . . yes
    libupsclient  . . . . yes
    libvarnish  . . . . . yes
    libvirt . . . . . . . yes
    libxenctrl  . . . . . yes
    libxml2 . . . . . . . yes
    libxmms . . . . . . . no
    libyajl . . . . . . . yes
    oracle  . . . . . . . no (ORACLE_HOME is not set)
    protobuf-c  . . . . . yes
    protoc 3  . . . . . . no

  Features:
    daemon mode . . . . . yes
    debug . . . . . . . . no

  Bindings:
    perl  . . . . . . . . yes (PREFIX=NONE)

  Modules:
    aggregation . . . . . yes
    amqp    . . . . . . . yes
    amqp1   . . . . . . . yes
    apache  . . . . . . . yes
    apcups  . . . . . . . yes
    apple_sensors . . . . no
    aquaero . . . . . . . no (libaquaero5.h not found)
    ascent  . . . . . . . yes
    barometer . . . . . . yes
    battery . . . . . . . yes
    bind  . . . . . . . . yes
    ceph  . . . . . . . . yes
    cgroups . . . . . . . yes
    chrony. . . . . . . . yes
    conntrack . . . . . . yes
    contextswitch . . . . yes
    cpu . . . . . . . . . yes
    cpufreq . . . . . . . yes
    cpusleep  . . . . . . yes
    csv . . . . . . . . . yes
    curl  . . . . . . . . yes
    curl_json . . . . . . yes
    curl_xml  . . . . . . yes
    dbi . . . . . . . . . yes
    df  . . . . . . . . . yes
    disk  . . . . . . . . yes
    dns . . . . . . . . . yes
    dpdkevents. . . . . . yes
    dpdkstat  . . . . . . yes
    drbd  . . . . . . . . yes
    email . . . . . . . . yes
    entropy . . . . . . . yes
    ethstat . . . . . . . yes
    exec  . . . . . . . . yes
    fhcount . . . . . . . yes
    filecount . . . . . . yes
    fscache . . . . . . . yes
    gmond . . . . . . . . yes
    gps . . . . . . . . . yes
    gpu_nvidia  . . . . . no
    grpc  . . . . . . . . no (protoc3 not found)
    hddtemp . . . . . . . yes
    hugepages . . . . . . yes
    intel_pmu . . . . . . no (jevents.h not found)
    intel_rdt . . . . . . yes
    interface . . . . . . yes
    ipc . . . . . . . . . yes
    ipmi  . . . . . . . . yes
    iptables  . . . . . . yes
    ipvs  . . . . . . . . yes
    irq . . . . . . . . . yes
    java  . . . . . . . . no (jar not found)
    load  . . . . . . . . yes
    logfile . . . . . . . yes
    log_logstash  . . . . yes
    lpar  . . . . . . . . no (AIX only)
    lua . . . . . . . . . yes
    lvm . . . . . . . . . yes
    madwifi . . . . . . . yes
    match_empty_counter . yes
    match_hashed  . . . . yes
    match_regex . . . . . yes
    match_timediff  . . . yes
    match_value . . . . . yes
    mbmon . . . . . . . . yes
    mcelog  . . . . . . . yes
    md  . . . . . . . . . yes
    memcachec . . . . . . yes
    memcached . . . . . . yes
    memory  . . . . . . . yes
    mic . . . . . . . . . no (MicAccessApi not found)
    modbus  . . . . . . . yes
    mqtt  . . . . . . . . yes
    multimeter  . . . . . yes
    mysql . . . . . . . . yes
    netapp  . . . . . . . no (netapp_api.h not found)
    netlink . . . . . . . yes
    network . . . . . . . yes
    nfs . . . . . . . . . yes
    nginx . . . . . . . . yes
    notify_desktop  . . . yes
    notify_email  . . . . yes
    notify_nagios . . . . yes
    ntpd  . . . . . . . . yes
    numa  . . . . . . . . yes
    nut . . . . . . . . . yes
    olsrd . . . . . . . . yes
    onewire . . . . . . . yes
    openldap  . . . . . . yes
    openvpn . . . . . . . yes
    oracle  . . . . . . . no (ORACLE_HOME is not set)
    ovs_events  . . . . . yes
    ovs_stats . . . . . . yes
    pcie_errors . . . . . yes
    perl  . . . . . . . . yes
    pf  . . . . . . . . . no
    pinba . . . . . . . . yes
    ping  . . . . . . . . yes
    postgresql  . . . . . yes
    powerdns  . . . . . . yes
    processes . . . . . . yes
    protocols . . . . . . yes
    python  . . . . . . . no
    redis . . . . . . . . yes
    routeros  . . . . . . no (routeros_api.h not found)
    rrdcached . . . . . . yes
    rrdtool . . . . . . . yes
    sensors . . . . . . . yes
    serial  . . . . . . . yes
    sigrok  . . . . . . . no (pkg-config could not find libsigrok)
    smart . . . . . . . . yes
    snmp  . . . . . . . . yes
    snmp_agent  . . . . . yes
    statsd  . . . . . . . yes
    swap  . . . . . . . . yes
    synproxy  . . . . . . yes
    syslog  . . . . . . . yes
    table . . . . . . . . yes
    tail_csv  . . . . . . yes
    tail  . . . . . . . . yes
    tape  . . . . . . . . no
    target_notification . yes
    target_replace  . . . yes
    target_scale  . . . . yes
    target_set  . . . . . yes
    target_v5upgrade  . . yes
    tcpconns  . . . . . . yes
    teamspeak2  . . . . . yes
    ted . . . . . . . . . yes
    thermal . . . . . . . yes
    threshold . . . . . . yes
    tokyotyrant . . . . . yes
    turbostat . . . . . . yes
    unixsock  . . . . . . yes
    uptime  . . . . . . . yes
    users . . . . . . . . yes
    uuid  . . . . . . . . yes
    varnish . . . . . . . yes
    virt  . . . . . . . . yes
    vmem  . . . . . . . . yes
    vserver . . . . . . . yes
    wireless  . . . . . . yes
    write_graphite  . . . yes
    write_http  . . . . . yes
    write_kafka . . . . . yes
    write_log . . . . . . yes
    write_mongodb . . . . yes
    write_prometheus. . . yes
    write_redis . . . . . yes
    write_riemann . . . . yes
    write_sensu . . . . . yes
    write_stackdriver . . yes
    write_syslog . .  . . yes
    write_tsdb  . . . . . yes
    xencpu  . . . . . . . yes
    xmms  . . . . . . . . no
    zfs_arc . . . . . . . yes
    zone  . . . . . . . . no
    zookeeper . . . . . . yes

Closing Throughts: These notes / helper tools are obviously specific to Ubuntu / Debian, but I believe that it would be useful to have simillar set of build asssistance tools for other platforms (RHEL, FreeBSD & Solaris as each of these has some OS specific plug-ins), as building upsteam collectd is not simple. I have suggested this within collectd pull request comments.


My First collectd Pull Request: https://github.com/collectd/collectd/pull/3153 a small step for open source, but a big step for an "enterprise architect" ;-)