Python Performance Tips

python-logo-master-flat-750

Python is a very powerful language that is focused on clear code readability.  It allows for very powerful programs to be done in fewer lines of code than traditional languages (C, C++, Java). However, this powerful syntax can, at times, come with some performance hits.

This post will analyze common programming operations and how one might go about coding these operations in Python.  We will then take a deeper look at these operations and how one might improve the performance using a different syntax mechanism.

String Building

In programming, it is quite common to need to build strings from multiple variables.  What most beginners to Python would do is use the  ‘+’ operator to concatenate strings.

head = '<head><title>My Page</title><link src="style.css" rel="stylesheet"/></head>'
body = '<body><h1>Hello, World</h1></body>'
out = '<html>' + head + body + '</html>'

So, why is this bad?

Well, strings in Python are immutable, meaning that they cannot be modified once created.  This is nice because they can be used as keys in dictionaries and individual copies can be shared across multiple variable bindings.  The downside is that any manipulation requires the creation of a new string.

So, in the example above, each ‘+’ operator is creating a new string.  If you are using a lot of these in your code with potentially large strings, then this simple mistake can really affect your performance. The solution is simple, however. Instead of the ‘+’ operator, build your strings with substitution.

out = '<html>%s%s</html>' % (head, body)

Alternatively, if you prefer, you can use the dictionary substitution which is cleaner looking to some programmers.

out = '<html>%(head)s%(body)s</html>' % locals()</pre>

 

It is also common to build a delineated string from a list.  For example

s = ''
fruits = ['orange','strawberry','banana','apple']
for fruit in fruits:
    s += fruit + ','
print s[:-1] # prints 'orange,strawberry,banana,apple'

As we learned before, the ‘+’ operator is bad, so how can we avoid it?  Well, the simple answer is just use a join

s = ','.join(fruits)
print s #prints 'orange,strawberry,banana,apple'

Not only is it more concise, but it is way more efficient!

Loops

Loops themselves are not necessarily bad. However, doing seemingly simple operations over each loop item (for loop) can often provide worse performance than you would expect. Here is a simple program that will take all items in a list and make them uppercase

words = ['hello','world']
newlist = []
for word in words:
    newlist.append(word.upper())
print newlist #prints ['HELLO','WORLD']

Seems harmless doesn’t it? Well, fact of the matter is that this isn’t necessarily bad, but it could be much better.  We can leverage list comprehensions (executes in C and can be orders of magnitude faster) to speed this up however.

newlist = [s.upper() for s in oldlist]
print newlist #prints ['HELLO','WORLD']

Additionally, if you would just want an iterable object that doesn’t have the overhead of generating the entire list at once, you can use a generator.

iterator = (s.upper() for s in oldlist)

 

Putting it All Together

Let’s say we want to generate a comma separated string of the upper case version of a list.

Without knowledge of Python performance, one might write the following:

mylist = ['abc','def','ghi']

newlist = []
for item in mylist:
    newlist.append(item.upper())

s = ''
for item in newlist:
    s += item + ','
print s[:-1]
#prints 'ABC,DEF,GHI'

However, the efficient, and much more concise version is:

mylist = ['abc','def','ghi']
print ','.join([s.upper() for s in mylist])
#prints 'ABC,DEF,GHI'

The efficient version replaces one loop with the join call which removes the multiple strings inefficiency from using the ‘+’ operator.  The list comprehension method is then used to reduce the time it takes to create the new list of upper case words.

How much faster is it? Here are some benchmarks using a list of strings. Each string has exactly three characters that are lower case and will be converted to upper case.  The y-axis is time to run, the x-axis is the number of items in the list, the blue bars indicate the inefficient algorithm, and the orange bars indicate the efficient algorithm.

As you can see, the 2nd algorithm is much more efficient.  Factor this in over an entire program and you can really increase the performance!

Benchmark script is below. It uses the timeit decorator (I have it in a file metrics.py) script from this post: Python Timing Decorator

from metrics import timeit

mylist = []
print "How many items to put into list: ",
num = raw_input()

for i in range(int(num)):
    mylist.append('abc')

@timeit(loops=1000)
def bad():
    newlist = []
    for item in mylist:
        newlist.append(item.upper())

    s = ''
    for item in newlist:
        s += item + ','
    return s[:-1]

@timeit(loops=1000)
def good():
    return ','.join([s.upper() for s in mylist])

bad()
good()

 

Share on Facebook+1Share on Twitter Share
Posted in Programming Tagged with: , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">