I recently asked on Twitter if it was possible to tell cProfile to start recording in the middle of a run, rather when a function first starts.
In my particular case, I have a function (generator, actually) that runs beautifully fast for the first 2.3 million items it returns, but then hits some strange bottleneck. I tried debugging with pdb, but on data sets as large and as complicated as I'm dealing with, I couldn't quite figure out what was going on.
My friend Chris pointed out that the runcall method of
Profile has references to
disable methods. I've never seen these before, and to be frank, I've never directly used the
Profile class. The
run function of the
cProfile module has always been good enough for me.
With a bit of playing around, I was able to use these methods to do what I needed. In this example,
chains is a generator. I simply disable the profiler immediately, and start it up at the point where I know things go wrong.
from cProfile import Profile profiler = Profile() def write_chains(): chain_count = 0 for chain in chains: chain_count += 1 print chain_count if chain_count == 2305838: profiler.enable() try: write_chains() except KeyboardInterrupt: profiler.dump_stats('prof_output2')
Chris pointed out that I don't need to wrap
write_chains() in a call to
run(), which I was originally doing. Also, that
disable() to be called, so I don't need to worry about that either.