carbon-cache supervisord FATAL start

Getting carbon-cache working with supervisord is a common thing you need to do when setting up Graphite or even the lighter weight graphite-api projects. Lots of the tutorials out there explain how to get this setup and most use supervisor to handle running carbon-cache.py.

When installing this recently via SaltStack, I ran into a peculiar error:

root@logs:/opt/statsd-0.7.2# supervisorctl status
carbon  FATAL  Exited too quickly (process log may have details)

The supervisor logs weren’t much help either:

carbon-cache (instance a) is already running with pid 17044
carbon-cache (instance a) is already running with pid 17044
carbon-cache (instance a) is already running with pid 17044
carbon-cache (instance a) is already running with pid 17044

So it turns out that is a normal situation with supervisor working with scripts that daemonize themselves. Don’t get me wrong, carbon-cache is started and running just perfectly when you see this fatal error in your supervisor status. It’s just supervisor can’t tell that it’s running correctly. To correct this situation you need to run carbon-cache.py with the —debug flag enabled like so:

[program:carbon]
command=/opt/graphite/bin/carbon-cache.py --debug start

Hopefully this help some of you to more quickly diagnose this issue that I ran into today.

Posted 10 November 2014

Frank Wiles

Founder of REVSYS and former President of the Django Software Foundation. Expert in building, scaling and maintaining complex web applications. Want to reach out? Contact me here or use the social links below.