I'm not sure how this happened but I somehow managed to get pulseaudio to consume about 5.5GB of RAM while nothing was playing any sound.