cProfile with imports

2024/11/14 20:09:49

I'm currently in the process of learn how to use cProfile and I have a few doubts.

I'm currently trying to profile the following script:

import timedef fast():print("Fast!")def slow():time.sleep(3)print("Slow!")def medium():time.sleep(0.5)print("Medium!")fast()
slow()
medium()

I execute the command python -m cProfile test_cprofile.py and I have the following result:

Fast!
Slow!
Medium!7 function calls in 3.504 secondsOrdered by: standard namencalls  tottime  percall  cumtime  percall filename:lineno(function)1    0.000    0.000    3.504    3.504 test_cprofile.py:1(<module>)1    0.000    0.000    0.501    0.501 test_cprofile.py:10(medium)1    0.000    0.000    0.000    0.000 test_cprofile.py:3(fast)1    0.000    0.000    3.003    3.003 test_cprofile.py:6(slow)1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}2    3.504    1.752    3.504    1.752 {time.sleep}

However, when I edit the script with a pylab import for example (import pylab) on the top, the output of cProfile is very large. I tried to limit the number of lines using python -m cProfile test_cprofile.py | head -n 10 however I receive the following error:

Traceback (most recent call last):
File "/home/user/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/user/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/user/anaconda/lib/python2.7/cProfile.py", line 199, in <module>
main()
File "/home/user/anaconda/lib/python2.7/cProfile.py", line 192, in main
runctx(code, globs, None, options.outfile, options.sort)
File "/home/user/anaconda/lib/python2.7/cProfile.py", line 56, in runctx
result = prof.print_stats(sort)
File "/home/user/anaconda/lib/python2.7/cProfile.py", line 81, in print_stats
pstats.Stats(self).strip_dirs().sort_stats(sort).print_stats()
File "/home/user/anaconda/lib/python2.7/pstats.py", line 360, in print_stats
self.print_line(func)
File "/home/user/anaconda/lib/python2.7/pstats.py", line 438, in print_line
print >> self.stream, c.rjust(9),
IOError: [Errno 32] Broken pipe

Can someone help what is the correct procedure to situations similar with this one, where we have an import pylab or another module that generates such high output information on cProfile?

Answer

I don't know of any way to do the selective profiling like you want by running the cProfile module directly from the command line like you're doing.

However, you can do it by modifying the your code to explicitly import the module, but you'll have to do everything yourself. Here's how that might be done to your example code:

(Note: The following code is compatible with both Python 2 and 3.)

from cProfile import Profile
from pstats import Stats
prof = Profile()prof.disable()  # i.e. don't time imports
import time
prof.enable()  # profiling back ondef fast():print("Fast!")def slow():time.sleep(3)print("Slow!")def medium():time.sleep(0.5)print("Medium!")fast()
slow()
medium()prof.disable()  # don't profile the generation of stats
prof.dump_stats('mystats.stats')with open('mystats_output.txt', 'wt') as output:stats = Stats('mystats.stats', stream=output)stats.sort_stats('cumulative', 'time')stats.print_stats()

mystats_output.txt file's contents afterwards:

Sun Aug 02 16:55:38 2015    mystats.stats6 function calls in 3.522 secondsOrdered by: cumulative time, internal timencalls  tottime  percall  cumtime  percall filename:lineno(function)2    3.522    1.761    3.522    1.761 {time.sleep}1    0.000    0.000    3.007    3.007 cprofile-with-imports.py:15(slow)1    0.000    0.000    0.515    0.515 cprofile-with-imports.py:19(medium)1    0.000    0.000    0.000    0.000 cprofile-with-imports.py:12(fast)1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Update:

You can make enabling the profiling a little easier by deriving your own Profile class with a context manager method to automate things. Instead of adding a method with a name like enable_profiling() to do this, I've implemented it so that you can just call the class instance in a with statement. Profiling will automatically be turned off whenever the context controlled by the with statement is exited.

Here's the class:

from contextlib import contextmanager
from cProfile import Profile
from pstats import Statsclass Profiler(Profile):""" Custom Profile class with a __call__() context manager method toenable profiling."""def __init__(self, *args, **kwargs):super(Profile, self).__init__(*args, **kwargs)self.disable()  # Profiling initially off.@contextmanagerdef __call__(self):self.enable()yield  # Execute code to be profiled.self.disable()

Using it instead of a stock Profile object would look something like this:

profiler = Profiler()  # Create class instance.import time  # Import won't be profiled since profiling is initially off.with profiler():  # Call instance to enable profiling.def fast():print("Fast!")def slow():time.sleep(3)print("Slow!")def medium():time.sleep(0.5)print("Medium!")fast()slow()medium()profiler.dump_stats('mystats.stats')  # Stats output generation won't be profiled.with open('mystats_output.txt', 'wt') as output:stats = Stats('mystats.stats', stream=output)stats.strip_dirs().sort_stats('cumulative', 'time')stats.print_stats()# etc...

Since it's a Profile subclass, all the base class' methods, such as dump_stats() are all still available for use as shown.

You could, of course, take it further and add e.g. a method to generate the stats and format them in some customized way.

https://en.xdnf.cn/q/71824.html

Related Q&A

Python AWS Lambda 301 redirect

I have a lambda handler written in Python and I wish to perform a 301 redirect to the browser. I have no idea how I can configure the response Location header (or the status code) -- the documentation …

Google Authenticator code does not match server generated code

BackgroundIm currently working on a two-factor authentication system where user are able to authenticate using their smartphone. Before the user can make use of their device they need to verify it firs…

Gekko Non-Linear optimization, object type error in constraint function evaluating if statement

Im trying to solve a non-linear optimization problem. Ive duplicated my issue by creating the code below. Python returns TypeError: object of type int has no len(). How can I include an IF statement in…

Store large dictionary to file in Python

I have a dictionary with many entries and a huge vector as values. These vectors can be 60.000 dimensions large and I have about 60.000 entries in the dictionary. To save time, I want to store this aft…

Python: override __str__ in an exception instance

Im trying to override the printed output from an Exception subclass in Python after the exception has been raised and Im having no luck getting my override to actually be called.def str_override(self):…

How hide/show a field upon selection of a radio button in django admin?

models.pyfrom django.db import models from django.contrib.auth.models import UserSTATUS_CHOICES = ((1, Accepted),(0, Rejected),) class Leave(models.Model):----------------status = models.IntegerField(c…

format/round numerical legend label in GeoPandas

Im looking for a way to format/round the numerical legend labels in those maps produced by .plot() function in GeoPandas. For example:gdf.plot(column=pop2010, scheme=QUANTILES, k=4)This gives me a lege…

Python pickle crash when trying to return default value in __getattr__

I have a dictionary like class that I use to store some values as attributes. I recently added some logic(__getattr__) to return None if an attribute doesnt exist. As soon as I did this pickle crashe…

How to download google source code for android

As you know, there is a list of several hundred projects in https://android.googlesource.com/. Id like to download them all in windows machine. According to Googles document,To install, initialize, and…

Compute on pandas dataframe concurrently

Is it feasible to do multiple group-wise calculation in dataframe in pandas concurrently and get those results back? So, Id like to compute the following sets of dataframe and get those results one-by…