perfmetrics

The perfmetrics package provides a simple way to add software performance metrics to Python libraries and applications. Use perfmetrics to find the true bottlenecks in a production application.

The perfmetrics package is a client of the Statsd daemon by Etsy, which is in turn a client of Graphite (specifically, the Carbon daemon). Because the perfmetrics package sends UDP packets to Statsd, perfmetrics adds no I/O delays to applications and little CPU overhead. It can work equally well in threaded (synchronous) or event-driven (asynchronous) software.

Complete documentation is hosted at https://perfmetrics.readthedocs.io

Usage

Use the @metric and @metricmethod decorators to wrap functions and methods that should send timing and call statistics to Statsd. Add the decorators to any function or method that could be a bottleneck, including library functions.

Caution

These decorators are generic and cause the actual function signature to be lost, replaced with *args, **kwargs. This can break certain types of introspection, including zope.interface validation. As a workaround, setting the environment variable PERFMETRICS_DISABLE_DECORATOR before importing perfmetrics or code that uses it will cause @perfmetrics.metric, @perfmetrics.metricmethod, @perfmetrics.Metric(...) and @perfmetrics.MetricMod(...) to return the original function unchanged.

Sample:

from perfmetrics import metric
from perfmetrics import metricmethod

@metric
def myfunction():
    """Do something that might be expensive"""

class MyClass(object):
    @metricmethod
    def mymethod(self):
        """Do some other possibly expensive thing"""

Next, tell perfmetrics how to connect to Statsd. (Until you do, the decorators have no effect.) Ideally, either your application should read the Statsd URI from a configuration file at startup time, or you should set the STATSD_URI environment variable. The example below uses a hard-coded URI:

from perfmetrics import set_statsd_client
set_statsd_client('statsd://localhost:8125')

for i in xrange(1000):
    myfunction()
    MyClass().mymethod()

If you run that code, it will fire 2000 UDP packets at port 8125. However, unless you have already installed Graphite and Statsd, all of those packets will be ignored and dropped. Dropping is a good thing: you don’t want your production application to fail or slow down just because your performance monitoring system is stopped or not working.

Install Graphite and Statsd to receive and graph the metrics. One good way to install them is the graphite_buildout example at github, which installs Graphite and Statsd in a custom location without root access.

Pyramid and WSGI

If you have a Pyramid app, you can set the statsd_uri for each request by including perfmetrics in your configuration:

config = Configuration(...)
config.include('perfmetrics')

Also add a statsd_uri setting such as statsd://localhost:8125. Once configured, the perfmetrics tween will set up a Statsd client for the duration of each request. This is especially useful if you run multiple apps in one Python interpreter and you want a different statsd_uri for each app.

Similar functionality exists for WSGI apps. Add the app to your Paste Deploy pipeline:

[statsd]
use = egg:perfmetrics#statsd
statsd_uri = statsd://localhost:8125

[pipeline:main]
pipeline =
    statsd
    egg:myapp#myentrypoint

Threading

While most programs send metrics from any thread to a single global Statsd server, some programs need to use a different Statsd server for each thread. If you only need a global Statsd server, use the set_statsd_client function at application startup. If you need to use a different Statsd server for each thread, use the statsd_client_stack object in each thread. Use the push, pop, and clear methods.

Graphite Tips

Graphite stores each metric as a time series with multiple resolutions. The sample graphite_buildout stores 10 second resolution for 48 hours, 1 hour resolution for 31 days, and 1 day resolution for 5 years. To produce a coarse grained value from a fine grained value, Graphite computes the mean value (average) for each time span.

Because Graphite computes mean values implicitly, the most sensible way to treat counters in Graphite is as a “hits per second” value. That way, a graph can produce correct results no matter which resolution level it uses.

Treating counters as hits per second has unfortunate consequences, however. If some metric sees a 1000 hit spike in one second, then falls to zero for at least 9 seconds, the Graphite chart for that metric will show a spike of 100, not 1000, since Graphite receives metrics every 10 seconds and the spike looks to Graphite like 100 hits per second over a 10 second period.

If you want your graph to show 1000 hits rather than 100 hits per second, apply the Graphite hitcount() function, using a resolution of 10 seconds or more. The hitcount function converts per-second values to approximate raw hit counts. Be sure to provide a resolution value large enough to be represented by at least one pixel width on the resulting graph, otherwise Graphite will compute averages of hit counts and produce a confusing graph.

It usually makes sense to treat null values in Graphite as zero, though that is not the default; by default, Graphite draws nothing for null values. You can turn on that option for each graph.

API Reference

Decorators

@metric

Notifies Statsd every time the function is called.

Sends both call counts and timing information. The name of the metric sent to Statsd is <module>.<function name>.

@metricmethod

Like @metric, but the name of the Statsd metric is <class module>.<class name>.<method name>.

class Metric(stat=None, rate=1, method=False, count=True, timing=True)

Bases: object

A decorator or context manager with options.

stat is the name of the metric to send; set it to None to use the name of the function or method. rate lets you reduce the number of packets sent to Statsd by selecting a random sample; for example, set it to 0.1 to send one tenth of the packets. If the method parameter is true, the default metric name is based on the method’s class name rather than the module name. Setting count to False disables the counter statistics sent to Statsd. Setting timing to False disables the timing statistics sent to Statsd.

Sample use as a decorator:

@Metric('frequent_func', rate=0.1, timing=False)
def frequent_func():
    "Do something fast and frequently."

Sample use as a context manager:

def do_something():
    with Metric('doing_something'):
        pass

If perfmetrics sends packets too frequently, UDP packets may be lost and the application performance may be affected. You can reduce the number of packets and the CPU overhead using the Metric decorator with options instead of metric or metricmethod. The decorator example above uses a sample rate and a static metric name. It also disables the collection of timing information.

When using Metric as a context manager, you must provide the stat parameter or nothing will be recorded.

Changed in version 3.0: When used as a decorator, set __wrapped__ on the returned object, even on Python 2.

Changed in version 3.0: When used as a decorator, the returned object has metric_timing, metric_count and metric_rate attributes that can be changed to alter its behaviour.

class MetricMod(format)

Bases: object

Decorator/context manager that modifies the name of metrics in context.

format is a format string such as ‘XYZ.%s’.

Functions

statsd_client()

Return the current StatsdClient for the thread.

Returns the thread-local client if there is one, or the global client if there is one, or None.

set_statsd_client(client_or_uri)

Set the global StatsdClient.

The client_or_uri can be a StatsdClient, a statsd:// URI, or None.

Note that when the perfmetrics module is imported, it looks for the STATSD_URI environment variable and calls set_statsd_client if that variable is found.

statsd_client_from_uri(uri)

Create and return perfmetrics.statsd.StatsdClient.

A typical URI is statsd://localhost:8125. An optional query parameter is prefix. The default prefix is an empty string.

StatsdClient Methods

Python code can send custom metrics by first getting the current IStatsdClient using the statsd_client() function. Note that statsd_client() returns None if no client has been configured.

interface IStatsdClient

Interface to communicate with a StatsD server.

Most of the methods below have optional rate, rate_applied, and buf parameters. The rate parameter, when set to a value less than 1, causes StatsdClient to send a random sample of packets rather than every packet. The rate_applied parameter, if true, informs StatsdClient that the sample rate has already been applied and the packet should be sent regardless of the specified sample rate.

If the buf parameter is a list, StatsdClient appends the packet contents to the buf list rather than send the packet, making it possible to send multiple updates in a single packet. Keep in mind that the size of UDP packets is limited (the limit varies by the network, but 1000 bytes is usually a good guess) and any extra bytes will be ignored silently.

close()

Release resources (sockets) held by this object.

New in version 3.0.

decr(stat, count=1, rate=1, buf=None, rate_applied=False)

Decrement a counter.

This is the opposite of incr().

gauge(stat, value, rate=1, buf=None, rate_applied=False)

Update a gauge value.

stat is the name of the metric to record and value is the new gauge value. A gauge represents a persistent value such as a pool size. Because gauges from different machines often conflict, a suffix is usually applied to gauge names; this may be done manually or with MetricMod.

incr(stat, count=1, rate=1, buf=None, rate_applied=False)

Increment a counter by count.

Note that Statsd clears all counter values every time it sends the metrics to Graphite, which usually happens every 10 seconds. If you need a persistent value, it may be more appropriate to use a gauge instead of a counter.

sendbuf(buf)

Send a UDP packet containing string lines.

buf is a sequence of strings.

set_add(stat, value, rate=1, buf=None, rate_applied=False)

Add a value to the set named by stat.

A StatsD set counts the unique occurrences of events (values) between flushes.

For example, if you wanted to count the number of different users logging in to an application within the sampling period, you could use something like:

def on_login(user_id):
    client.set_add("logged_in_users", user_id)

While this method accepts the rate parameter, it may be less useful here since the point is to let the StatsD server collect unique events automatically, and it can’t do that if some events are dropped, making it only an estimate.

New in version 3.1.0.

timing(stat, value, rate=1, buf=None, rate_applied=False)

Log timing information in milliseconds.

stat is the name of the metric to record and value is the timing measurement in milliseconds. Note that Statsd maintains several data points for each timing metric, so timing metrics can take more disk space than counters or gauges.

There are three implementations of this interface:

class StatsdClient(host='localhost', port=8125, prefix='')

Bases: object

Send packets to statsd.

Default implementation of perfmetrics.interfaces.IStatsdClient.

Derived from statsd.py by Steve Ivy <steveivy@gmail.com>.

close()

See perfmetrics.interfaces.IStatsdClient.close().

New in version 3.0.

decr(stat, count=1, rate=1, buf=None, rate_applied=False)

See perfmetrics.interfaces.IStatsdClient.decr().

gauge(stat, value, rate=1, buf=None, rate_applied=False)

See perfmetrics.interfaces.IStatsdClient.gauge().

incr(stat, count=1, rate=1, buf=None, rate_applied=False)

See perfmetrics.interfaces.IStatsdClient.incr().

sendbuf(buf)

See perfmetrics.interfaces.IStatsdClient.sendbuf().

set_add(stat, value, rate=1, buf=None, rate_applied=False)

See perfmetrics.interfaces.IStatsdClient.set_add().

timing(stat, value, rate=1, buf=None, rate_applied=False)

See perfmetrics.interfaces.IStatsdClient.timing().

class StatsdClientMod(wrapped, format)

Bases: object

Wrap StatsdClient, modifying all stat names in context.

Changed in version 3.0: The wrapped object’s attributes are now accessible on this object.

This object now uses __slots__.

class NullStatsdClient

Bases: object

No-op statsd client.

close()

Does nothing.

decr(stat, *args, **kw)

Does nothing.

gauge(stat, *args, **kw)

Does nothing.

incr(stat, *args, **kw)

Does nothing.

sendbuf(buf)

Does nothing

set_add(stat, value, *args, **kw)

Does nothing.

timing(stat, *args, **kw)

Does nothing.

Pyramid Integration

includeme(config)

Pyramid configuration hook: activate the perfmetrics tween.

A statsd_uri should be in the settings.

tween(handler, registry)

Pyramid tween that sets up a Statsd client for each request.

WSGI Integration

make_statsd_app(nextapp, _globals=None, statsd_uri='')

Create a WSGI filter app that sets up Statsd for each request.

If no statsd_uri is given, returns nextapp unchanged.

Changed in version 3.0: The returned app callable makes the statsd client that it uses available at the statsd_client attribute.

Testing

perfmetrics.testing provides a testing client for verifying StatsD metrics emitted by perfmetrics in the context of an instrumented application.

It’s easy to create a new client for use in testing:

>>> from perfmetrics.testing import FakeStatsDClient
>>> test_client = FakeStatsDClient()

This client exposes the same public interface as perfmetrics.statsd.StatsdClient. For example we can increment counters, set gauges, etc:

>>> test_client.incr('request_c')
>>> test_client.gauge('active_sessions', 320)

Unlike perfmetrics.statsd.StatsdClient, FakeStatsDClient simply tracks the statsd packets that would be sent. This information is exposed on our test_client both as the raw statsd packet, and for convenience this information is also parsed and exposed as Observation objects. For complete details see FakeStatsDClient and Observation.

>>> test_client.packets
['request_c:1|c', 'active_sessions:320|g']
>>> test_client.observations
[Observation(name='request_c', value='1', kind='c', sampling_rate=None), Observation(name='active_sessions', value='320', kind='g', sampling_rate=None)]

For validating metrics we provide a set of PyHamcrest matchers for use in test assertions:

>>> from hamcrest import assert_that
>>> from hamcrest import contains_exactly as contains
>>> from perfmetrics.testing.matchers import is_observation
>>> from perfmetrics.testing.matchers import is_gauge

We can use both strings and numbers (or any matcher) for the value:

>>> assert_that(test_client,
...     contains(is_observation('c', 'request_c', '1'),
...              is_gauge('active_sessions', 320)))
>>> assert_that(test_client,
...     contains(is_observation('c', 'request_c', '1'),
...              is_gauge('active_sessions', '320')))
>>> from hamcrest import is_
>>> assert_that(test_client,
...     contains(is_observation('c', 'request_c', '1'),
...              is_gauge('active_sessions', is_('320'))))

If the matching fails, we get a descriptive error:

>>> assert_that(test_client,
...     contains(is_gauge('request_c', '1'),
...              is_gauge('active_sessions', '320')))
Traceback (most recent call last):
...
AssertionError:
Expected: a sequence containing [(an instance of Observation and (an object with a property 'kind' matching 'g' and an object with a property 'name' matching 'request_c' and an object with a property 'value' matching '1')), (an instance of Observation and (an object with a property 'kind' matching 'g' and an object with a property 'name' matching 'active_sessions' and an object with a property 'value' matching '320'))]
       but: item 0: was Observation(name='request_c', value='1', kind='c', sampling_rate=None)

Reference

perfmetrics.testing

class FakeStatsDClient(prefix='')

Bases: StatsdClient

A mock statsd client that tracks sent statsd metrics in memory rather than pushing them over a socket. This class is a drop in replacement for perfmetrics.statsd.StatsdClient that collects statsd packets and Observation that are sent through the client.

Changed in version 3.1.0: Like the normal clients, this object is now always true, whether or not any observations have been sent.

__len__()

The number of metrics sent. This accounts for multi metric packets that may be sent.

clear()

Clears the statsd metrics that have been collected

iter_observations()

Iterates the Observations provided to this statsd client.

iter_packets()

Iterates the raw statsd packets provided to the statsd client.

Returns

Iterator of native strings.

property observations

A list of Observation objects collected by this client.

See also

iter_observations

property packets

A list of raw statsd packets collected by this client.

See also

iter_packets

OBSERVATION_KIND_COUNTER = 'c'

The statsd metric kind for Counters

OBSERVATION_KIND_GAUGE = 'g'

The statsd metric kind for Gauges

OBSERVATION_KIND_SET = 's'

The statsd metric kind for Sets

OBSERVATION_KIND_TIMER = 'ms'

The statsd metric kind for Timers

class Observation(name, value, kind, sampling_rate=None)

Bases: object

The representation of a single statsd metric.

kind = None

The statsd code for the type of metric. e.g. one of the METRIC_*_KIND constants

classmethod make(packet)

Creates a metric from the provided statsd packet.

Raises

ValueError – if packet is a multi metric packet or otherwise invalid.

classmethod make_all(packet)

Makes a list of metrics from the provided statsd packet.

Like make but supports multi metric packets

name = None

The metric name

sampling_rate = None

The rate with which this event has been sampled from (optional)

value = None

The value provided for the metric

perfmetrics.testing.matchers

is_counter(*, name, value, sampling_rate) → matcher

A hamcrest matcher validating the parts of a counter Observation.

See also

is_metric

is_gauge(*, name, value, sampling_rate) → matcher

A hamcrest matcher validating the parts of a gauge Observation

See also

is_metric

is_observation(*, kind, name, value, sampling_rate) → matcher

A hamcrest matcher that validates the specific parts of a Observation. All arguments are optional and can be provided by name or position.

Parameters

• kind (str) – A hamcrest matcher or string that matches the kind for this metric

• name (str) – A hamcrest matcher or string that matches the name for this metric

• value (str) – A hamcrest matcher or string that matches the value for this metric

• sampling_rate (float) – A hamcrest matcher or number that matches the sampling rate this metric was collected with

is_set(*, name, value, sampling_rate) → matcher

A hamcrest matcher validating the parts of a set Observation

See also

is_metric

is_timer(*, name, value, sampling_rate) → matcher

A hamcrest matcher validating the parts of a timer Observation

See also

is_metric

CHANGES

4.0.1 (unreleased)

• Nothing changed yet.

4.0.0 (2023-06-22)

• Drop support for obsolete Python versions, including Python 2.7 and 3.6.

• Add support for Python 3.12.

3.3.0 (2022-09-25)

• Stop accidentally building manylinux wheels with unsafe math optimizations.

• Add support for Python 3.11.

NOTE: This will be the last major release to support legacy versions of Python such as 2.7 and 3.6. Some such legacy versions may not have binary wheels published for this release.

3.2.0.post0 (2021-09-28)

• Add Windows wheels for 3.9 and 3.10.

3.2.0 (2021-09-28)

• Add support for Python 3.10.

• Drop support for Python 3.5.

• Add aarch64 binary wheels.

3.1.0 (2021-02-04)

• Add support for Python 3.8 and 3.9.

• Move to GitHub Actions from Travis CI.

• Support PyHamcrest 1.10 and later. See issue 26.

• The FakeStatsDClient for testing is now always true whether or not any observations have been seen, like the normal clients. See issue.

• Add support for StatsD sets, counters of unique events. See PR 30.

3.0.0 (2019-09-03)

• Drop support for EOL Python 2.6, 3.2, 3.3 and 3.4.

• Add support for Python 3.5, 3.6, and 3.7.

• Compile the performance-sensitive parts with Cython, leading to a 10-30% speed improvement. See https://github.com/zodb/perfmetrics/issues/17.

• Caution: Metric names are enforced to be native strings (as a result of Cython compilation); they’ve always had to be ASCII-only but previously Unicode was allowed on Python 2. This is usually automatically the case when used as a decorator. On Python 2 using from __future__ import unicode_literals can cause problems (raising TypeError) when manually constructing Metric objects. A quick workaround is to set the environment variable PERFMETRICS_PURE_PYTHON before importing perfmetrics.

• Make decorated functions and methods configurable at runtime, not just compile time. See https://github.com/zodb/perfmetrics/issues/11.

• Include support for testing applications instrumented with perfmetrics in perfmetrics.testing. This was previously released externally as nti.fakestatsd. See https://github.com/zodb/perfmetrics/issues/9.

• Read the PERFMETRICS_DISABLE_DECORATOR environment variable when perfmetrics is imported, and if it is set, make the decorators @metric, @metricmethod, @Metric(...) and @MetricMod(...) return the function unchanged. This can be helpful for certain kinds of introspection tests. See https://github.com/zodb/perfmetrics/issues/15

2.0 (2013-12-10)

• Added the @MetricMod decorator, which changes the name of metrics in a given context. For example, @MetricMod('xyz.%s') adds a prefix.

• Removed the “gauge suffix” feature. It was unnecessarily confusing.

• Timing metrics produced by @metric, @metricmethod, and @Metric now have a “.t” suffix by default to avoid naming conflicts.

1.0 (2012-10-09)

• Added ‘perfmetrics.tween’ and ‘perfmetrics.wsgi’ stats for measuring request timing and counts.

0.9.5 (2012-09-22)

• Added an optional Pyramid tween and a similar WSGI filter app that sets up the Statsd client for each request.

0.9.4 (2012-09-08)

• Optimized the use of reduced sample rates.

0.9.3 (2012-09-08)

• Support the STATSD_URI environment variable.

0.9.2 (2012-09-01)

• Metric can now be used as either a decorator or a context manager.

• Made the signature of StatsdClient more like James Socol’s StatsClient.

0.9.1 (2012-09-01)

• Fixed package metadata.

0.9 (2012-08-31)

• Initial release.

Indices and tables

• Index

• Module Index

• Search Page