user.p0d.org

bits release in progress

Python2.6 performances

Tags python

Hasty conclusion

Through the tests i came to the conclusion that function call in Python does have a signifiant impact on performances; it is for the best if you can reduce their number (especially in loops, list comprehension...).

Howto

import timeit
timeit.timeit('"%(foo)s%(bar)s" % locals()', 'foo="foo";bar="bar"')
# or
timeit.timeit('func()', 'from __main__ import func')
python -mtimeit -s'import random' \
   -s'tuple(random.randint(32, 126) for x in range(1000))'

Also with profile:

import profile
profile.run('map(str, range(256))')
Legend
%:
time relation (in percent) to the "best" solution
f:
number of function call as seen in profile.run
Δ:
really rounded up difference in seconds between the best and the worst scenarios on my machine at a given time (ie: 1e-6 mean it would take one million calls to make one second difference)

Sorting

% f data=zip(range(100), range(100)[::-1])
97 7 sorted(data, key=k) # k = operator.item...
100 7 sorted(data, key=operator.itemgetter(1))
171 107 sorted(data, key=lambda x:x[1])

object.sort() and sorted() have similar performances.


String concatenation and format

% f operation
100 3 '%s%s' % (a, b)
101 4 ''.join((a, b))
259 3 '%(a)s%(b)s' % {'a': a, 'b': b}
429 4 '%(a)s%(b)s' % locals()

Δ: 2.93e-6

   
103 '%s%s' % (a, b)
100 ''.join((a, b))
137 '%(a)s%(b)s' % {'a': a, 'b': b}
208 '%(a)s%(b)s' % locals()
31 a+b"

Δ: 6.05e-8


List convertion

% f data = range(256)
100 5 map(str, data)
150 4 [str(x) for x in data]
174 4 for i,v in enumerate(data): data[i]=str(v) # inplace!
203 260 for x in data: n.append(str(x))
264 4 for x in data: n+= [str(x)]

regexp

Compilation and cache is really time consuming.

% data = open(...).readlines(); p = '(:[x0-9]+:0:|:[^:]*root)'
99 filter(lambda x:r.search(x), data) # r = sre_compile.compile(p)
100 filter(lambda x:r.search(x), data) # r = re.compile(p)
203 filter(lambda x:re.search(p, x), data)
re vs. fnmatch

By the way fnmatch use re...

% f data = os.listdir(...)
100 271 [r.match(x) for x in data] # r=re.compile(...)
435 1063 [re.match('..*.o.cmd', x) for x in data]
445 1063 [re.match('..*.o.cmd$', x) for x in data]
581 1324 [fnmatch.fnmatch(x, '.*.o.cmd') for x in data]

os.stat and memoizer

While testing if it's really worth to cache os.stat results, i took the opportunity to do some memoization benchmarks (it is more a lazy resolution than a memoization).

The differences tend to reduce as the number of call get lower (timeit number argument) but the order stay the same.

%  
22 f.override
100 f.cached()
119 f.cache
699 f.stat()
758 f.s
foo = '''
import os
class Foo:
   __stat = None
   path   = None
   def __init__(self, path):
      self.path = path
   def cached(self):
      if not self.__stat:
         self.__stat = os.stat(self.path)
      return self.__stat
   cache = property(cached)
   @property
   def override(self):
      self.override = os.stat(self.path)
      return self.override
   def stat(self):
      return os.stat(self.path)
   s = property(stat)

f = Foo('.')
assert f.override == f.cached() == f.cache == f.stat() == f.s
'''

for d in sorted(
   ((timeit.timeit(k, foo),k) for k in ('f.override', 'f.cached()',
   'f.cache', 'f.stat()', 'f.s')),
   key=operator.itemgetter(0)
): print '%21.18f %s'%d

dictionary creation

%  
100 d={}nfor i in data: d[i]=i
167 d=dict((i,i) for i in data)

Simples operations

while not the end-of-the-universe
%  
73 while "x": ...
74 while 1: ...
100 while True: ...
115 while type: ...
131 while ((),()): ...
156 while [[]]: ...
for d in sorted(
   ((timeit.timeit('c=10000\nwhile %s:\n
   if not c:break\n c-=1'%(k,)),k) for k in ops),
   key=operator.itemgetter(0)
): print '%21.18f %s'%d
if true, if false
% if true
42 if "x": ...
72 if 1: ...
100 if True: ...
124 if type: ...
166 if ((),()): ...
220 if [[]]: ...
% if false
37 if "": ...
62 if 0: ...
86 if None: ...
90 if (): ...
100 if False: ...
113 if []: ...

Δ: 3.2e-7

ops = (1, True, '"x"', [[]], ((),()), 'type')
ops = (0, False, '""', [], (), None)
for d in sorted(
   ((timeit.timeit('if %s:pass'%(k,), number=1000),k) for k in ops),
   key=operator.itemgetter(0)
): print '%21.18f %s'%d
if-then-else or ternary operator
%  
100 1 if True else 2;1 if False else 2
91 (2,1)[True];(2,1)[False]
87 True and 1 or 2;False and 1 or 2

Δ: 1.7e-7

shift or mul?
%  
99 [x*y for x in data for y in (4, 256, 65536, 16777216, 4294967296L)]
100 [x<<y for x in data for y in (2,8,16,24,32)]

Δ: 6.1e-8


map-reduce (and lambda) vs list comprehension

At least PEP 289 encourage to use list comprehension instead of map-reduce; is it really for the best?

Usage of map is also seen at the list convertion section.

The following example is an IPv4 to long; but such technique might prove useful to RGB color representation in hexa (i.e.: '#%x' % convert((0x11, 0x22, 0x33)) would yield #112233).

% data = (127,0,0,1)
100 reduce(lambda a,b: a*256 + b, data)
273 sum(v * (256**(len(data)-i-1)) for i,v in enumerate(data))
280 sum(v<<(8*(len(data)-i-1)) for i,v in enumerate(data) )

Δ: 1e-5

% type of the elements of the iterable if identic
72 reduce(lambda x,y:x if x.__class__ == y.__class__ else False, data)
79 type(reduce(lambda x,y:x if x.__class__ == y.__class__ else False, data))
91 type(reduce(lambda x,y:x if type(x) == type(y) else False, data))
100 reduce(lambda x,y:x if x == y else False, map(type,data))
116 reduce(lambda x,y:x if x == y else False, (type(x) for x in data))

Δ: 5.1e-6


locals() vs dir()

%  
100 def x(a,b,c): return min((a,b,c))
182 def x(a,b,c): return min(locals().itervalues())
222 def x(a,b,c): return min(locals().values())
379 def x(a,b,c): return min(dir())

Δ: 8.9e-6


Operators: binary vs. logicals

% data = ( (0, 0, 0), (1, 0, 0), (1, 1, 0), (1, 1, 1), (0, 1, 1), (0, 0, 1), (0, 1, 0), (1, 0, 1))
100 [(y,m,d) for y,m,d in data if (y and (m or not d))] # logical
132 [(y,m,d) for y,m,d in data if (y & (m | ~d))] # binary

Δ: 2.9e-6


DefaultDict

Is defaultdict worth it? Well, yes it seems.

100 defaultdict
109 plan B
232 plan A

Using dict, plan A:

d = {}
for x in range(1024):
   k = '%x'%(x%0x10)
   d[k] = d.get(k, [])+[x]

Using dict, plan B:

d = {}
for x in range(1024):
   k = '%x'%(x%0x10)
   if k in d: d[k].append(x)
   else: d[k] = [x]

Using defaultdict:

d = __import__('collections').defaultdict(list)
for x in range(1024):
   d['%x'%(x%0x10)].append(x)

Δ: 4.8e-6


Insert, append, reverse?

100 planA()
73 planB()
55 planC()
57 planD()
23 planE()

Δ: 1.4e-6

When creating a list and needing it reversed:

def planA(stop=100):
   d=[]
   for i in range(stop):
      d.insert(0, i)
   return d

def planB(stop=100):
   d=[]
   for i in range(stop):
      d.append(i)
   return list(reversed(d)) # do not return an iterator!

def planC(stop=100):
   d=[]
   for i in range(stop):
      d.append(i)
   d.reverse()
   return d

def planD(stop=100):
   d=[]
   for i in range(stop):
      d.append(i)
   return d[::-1]

def planE(stop=100):
   return [i for i in range(stop)][::-1]