Archive for the ‘Python’ Category

Shared post: Using SCons effectively

June 1, 2016

An overall interesting read to understand core concepts of SCons:

http://blog.bfitz.us/?p=1679

 

Advertisements

Profiling with Python: cProfile and RunSnakeRun

June 1, 2010

This is my favorite way to analyze the performance of my Python scripts:

  1. Call the script with cProfile:
    python -m cProfile -o profile.dat main.py
  2. Open the profiler data with RunSnakeRun:
    %PYTHON_ROOT%\scripts\runsnake.exe
    A nice GUI will start up, and you can open the profile.dat file in there.
  3. You have to install RunSnakeRun first, of course. You should choose the easy_install procedure I described here for the runsnake.exe to be installed in the scripts directory. You can find the current RunSnakeRun package on pypi.

    RunSnakeRun homepage

Libxml2 and Python on Windows

May 19, 2010

For the purpose of using XPath and XQuery out of my python scripts for test statistics generation, I decided to try libxml2. I mostly decided against ElementTree because their website told me the XPath subset does not support queries like:


count(//event[evalresult/text()="FALSE"][warnlevel/text()="1"])

Also, I hope to be faster with the C-based implementation of libxml2. For installation, I only had to download this pre-bundled lxml windows binary from the python package repository; it comes with libxml2 included.

The following code is enough to get the above count:

from lxml import etree
doc = etree.parse(filePath)
result = doc.xpath('//event[evalresult/text()="FALSE"][warnlevel/text()="1"]')
count = len(result)
count2 = doc.xpath(count('//event[evalresult/text()="FALSE"][warnlevel/text()="1"]')) # alternative

About XQuery: lxml serves as frontend for libxml2 and libxslt, neither of which support XQuery.

Further information:
lxml homepage
SketchPath, a good XPath expression evaluation software

Python Pitfalls: References to the same dictionary

May 18, 2010

Today, I tried to build a dictionary which contains the contents of a list as keys and empty dictionarys as values. First, I went with the method dict.fromkeys(seq, [value]), then I filled the sub-dictionaries in a for loop, like this:

mydict = dict.fromkeys(lstLabels, {})
for strLabel in dict.keys():
    mydict[strLabel][strLabel] = "x"

What I ended up with was that the sub-dictionaries referenced by the keys from lstLabels all referenced to the same dictionary instance. What this means is, that whenever you add something to dict[lstLabels[0]], it appears in dict[lstLabels[1]], too, and so on.

The correct way to accomplish what I wanted – different instances of subdirectories – was this:

    for strLabel in lstLabels:
        mydict[strLabel] = {}
        mydict[strLabel][strLabel] = "x"

Apparently, the dict.fromkeys() method does something like lstFail = [[] * 3]

Are there more elegant ways to create a dictionary with subdictionarys based on a list of strings than the one presented?

Installing Python packages with easy_install

May 17, 2010

Python comes with a nice tool called easy_install that enables you to install additional packages as easy as:

easy_install <package_name>

If you are behind a HTTP proxy server, this might fail unless you specify the proxy URL in the environment variable HTTP_PROXY like this:

set http_proxy=”http://www.myproxy.org&#8221;

However, when you are sitting behind a firewall (like we at my company are), you might still not be able to download any packages from the python package repository. Here’s a quick guide how to circumvent the issue:

  1. Create a folder on your local machine that will contain all packages you have to download.
  2. Go to http://pypi.python.org/simple/ and download the package you want to install into your local folder.
  3. In the scripts directory of your python installation, call:
    easy_install -H none -f d:\Backup\Programs\python-packages <package_name>
  4. When you disallow all foreign hosts with “-H none”, easy_install will not be able to fetch any packages your package depends on, so you have to resolve all dependency errors by repeating step 1 – 3 for the missing package if easy_install gives you a message like this:
    “No local packages or download links found for logilab-common>=0.49.0”

Another solution I found on the internet would be to install and use ASProxy, but I’m not going to try that for now.

See also: easy_install guide