ClickHouse Monitoring with Graphite
Apr 20, 2018
Monitoring is an important part of operating any software in production. ClickHouse is not an exception here. ClickHouse provides an introspection to various system metrics using tables in 'system' schema but they only show current or cumulative numbers. It is sufficient in order to understand what's going on with the system right now, but can not show time trends or what has been happening a minute ago. The proper monitoring solution needs to store metrics somewhere as well as provide visualization capabilities. ClickHouse does not have such a tool packaged, but there are several 3rd-party monitoring solutions that can be used. Graphite is one of the popular options, and it can be natively integrated with ClickHouse. In this article, we will describe how to configure Graphite for monitoring ClickHouse metrics.
Graphite installation is described in details in its manual Feel free to choose any type of installation.
Setup ClickHouse - Graphite integration
ClickHouse configuration for Graphite is specified in ClickHouse config file. Edit main config file
/etc/clickhouse-server/config.xml or add a separate
/etc/clickhouse-server/conf.d/graphite.xml, and append following section:
<graphite> <host>192.168.74.150</host> <port>2003</port> <timeout>0.1</timeout> <interval>60</interval> <root_path>one_min_cr_plain</root_path> <metrics>true</metrics> <events>true</events> <asynchronous_metrics>true</asynchronous_metrics> </graphite> <graphite> <host>192.168.74.150</host> <port>2003</port> <timeout>0.1</timeout> <interval>1</interval> <root_path>one_sec_cr_plain</root_path> <metrics>true</metrics> <events>true</events> <asynchronous_metrics>false</asynchronous_metrics> </graphite>
host– host where Graphite is running.
port– Carbon plain text receiver port (2003 is default in Graphite). In general, Graphite has multiple ports open, with the following default values:
2003carbon receiver - plaintext this one should receive data from ClickHouse
2004carbon receiver - pickle
2023carbon aggregator - plaintext
2024carbon aggregator - pickle
8080Graphite internal gunicorn port (without Nginx proxying).
interval– interval for sending data from ClickHouse, in seconds.
timeout– timeout for sending data, in seconds.
root_path– prefix used by Graphite.
metrics– should data from
system_tables-system.metricstable be sent.
events– should data from
system_tables-system.eventstable be sent.
asynchronous_metrics– should data from
system_tables-system.asynchronous_metricstable be sent.
<graphite> sections can be used for sending different data at different intervals.
NOTE, ClickHouse restart is required in order to activate graphite configuration.
Navigate to Graphite web monitoring tool in a browser as
http://host/dashboard in order to see Graphite metrics.
If everything is properly configured, and ClickHouse sends its data to Graphite, entries with prefixes should be displayed, as specified in ClickHouse's configuration, e.g.:
For all available metrics Graphite will display monitoring graphs.