Curious case of closure in Go and Python

At http://tour.golang.org/#39, I found the following sample code:

package main

import "fmt"

func adder() func(int) int {
	sum := 0
	return func(x int) int {
		sum += x
		return sum
	}
}

func main() {
	pos, neg := adder(), adder()
	for i := 0; i < 10; i++ {
		fmt.Println(
			pos(i),
			neg(-2*i),
		)
	}
}

Its document reads:

… functions are full closures. The adder function returns a closure. Each closure is bound to its own sum variable.

Take that into consideration, it might be easier to understand the execution result:

0 0
1 -2
3 -6
6 -12
10 -20
15 -30
21 -42
28 -56
36 -72
45 -90

To me, variable sum is similiar to ‘instance variable’, in Object-oriented’s terminology. However, in Python, things can be quite different.

def adder():
    sum = 0

    def f(x):
        sum += x
        return sum
    return f


def main():
    pos, neg = adder(), adder()
    for i in xrange(0, 10):
        print pos(i), neg(-2 * i)

The code looks roughly the same, but it will raise the following exception:

Traceback (most recent call last):
  File "closure.py", line 17, in 
    main()
  File "closure.py", line 13, in main
    print pos(i), neg(-2 * i)
  File "closure.py", line 5, in f
    sum += x
UnboundLocalError: local variable 'sum' referenced before assignment

This is because if sum is to be modified, Python must decide which variable to change. sum += x is the same as sum = sum + x, and sum = suggests it’s a local variable, since all variable is by default local in Python. Given that, expression sum + x can not be evaluated because sum, as a local variable, is still undefined here.

If the sum += x line is removed, and sum + x is returned directly, the result will be:

0 0
1 -2
2 -4
3 -6
4 -8
5 -10
6 -12
7 -14
8 -16
9 -18

It runs okay, but the result is wrong. Where does function f get the value of sum? If Python cannot find a variable in locals(), it will try to find it from the scope above it, i.e. function adder, and sum is indeed defined in it. The real Python equivelent of the Go program above will be:

class adder:
    def __init__(self):
        self.sum = 0
    def __call__(self, x):
        self.sum += x
        return self.sum


def main():
    pos, neg = adder(), adder()
    for i in xrange(0, 10):
        print pos(i), neg(-2 * i)


if __name__ == '__main__':
    main()

Functions are already first class objects in Python. Here we create a class that its instance behaves like a function, so it is a function because of duck typing.

Swap every windows between workspaces in awesome wm

awesome is the tiling windows manager I use daily and I just wrote a very useful (at least for me) function for it. Basically it swaps every client (similar to windows) between the current tag (similar to workspace) and the target tag. Normally I put terminal clients for my current task at tag#3, right next to my Emacs-only tag#2 so I can quickly switch between browser (#1), editor and terminals. Now if I want to switch to a different task I just need to swap in terminals from another tag while swap out current terminals to that tag.

-- i is the index of target tag in variable `tags'
function ()
   local screen = mouse.screen
   local from = client.focus:tags()[1]
   local to = tags[screen][i]
   if to then
       t = to:clients()
       for i, c in ipairs(from:clients()) do
           awful.client.movetotag(to, c)
       end
       for i, c in ipairs(t) do
           awful.client.movetotag(from, c)
       end
   end
end

I put this under

-- Bind all key numbers to tags.
-- Be careful: we use keycodes to make it works on any keyboard layout.
-- This should map on the top row of your keyboard, usually 1 to 9.
for i = 1, keynumber do
    globalkeys = awful.util.table.join(globalkeys,
        awful.key({ modkey }, "#" .. i + 9,
                  function ()
                        local screen = mouse.screen
                        if tags[screen][i] then
                            awful.tag.viewonly(tags[screen][i])
                        end
                  end),
        awful.key({ modkey, "Control" }, "#" .. i + 9,

in my copy of default rc.lua.

Multi-thread testing in Pyramid

If you want to do multi-thread testing in Pyramid, it probably won’t work the first time because request and registry are thread local, and things like get_renderer will call get_current_registry. When it happens in a thread, it won’t get the same value as it would have in the main thread.

So, here is a hack to address this:

import pyramid.threadlocal
from threading import Thread, Lock

candidates = [
    (self._test1, ()),
    (self._test2, ()),
    (self._test3, ()),
    ]

def random_func(pyramid_thread_locals):
    pyramid.threadlocal.manager.push(pyramid_thread_locals)
    time.sleep(random.random())  # 0 ~ 1 sec
    func, args = random.choice(candidates)
    func(*args)

pyramid_thread_locals = pyramid.threadlocal.manager.get()
threads = [Thread(target=random_func, args=(pyramid_thread_locals, ),)
           for i in range(100)]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

There is no guarantee that pyramid.threadlocal.manager will always be there. Even if it’s there, there’s no guarantee it can be used this way. So, this should only be considered as a temporary workaround.

Launching New Studio

JJ’s Studio is a software consultancy specialized in Android technologies. We are experienced in both system and application layers, plus several years of experience before that in Linux mobile phone development.

We have co-founded the well-known 0xlab.org, as well as created/contributed to several Free and Open Source Software projects, such as 0xdroid, 0xbench, OpenEmbedded, and OpenWrt.

Our services include:

  1. Android application: standard Android application that fits into the varieties of Android devices.
  2. Framework development: develop or integrate new features into device firmware in framework layer and below.
  3. Performance analysis: get more out of your hardware by profiling and tuning.

JJ’s Studio differentiates ourselves by the unique combination of knowledge in different layers of Android architecture. This enables us to work in different areas from developing a mobile application to delivering a complete device firmware.

Please also check my partner Julian’s article on this (in Traditional Chinese).

Contact:

John Lee
john@0xlab.org

(Traditional Chinese)

JJ’s Studio 為專精於 Android 相關科技的軟體顧問工作室。我們對於系統與應用層皆有經驗,並且在 Android 之前就有數年的 Linux 手機開發資歷。

我們與朋友共同創立了知名的 0xlab.org,並曾創立或貢獻給以下的開放原始碼專案:0xdroid, 0xbench, OpenEmbedded, OpenWrt.

我們的服務包括:

  1. Android 應用程式:著重於對不同 Android 裝置的相容與一致性。
  2. 框架層開發:將新功能開發或整合進 Android 框架或底層系統之中,直接包含在出廠韌體內。
  3. 效能分析:對系統剖析與調整,讓硬體發揮更大效能。

JJ’s Studio 的獨特競爭優勢來自於對 Android 系統各不同層面的理解。這讓我們可以勝任從應用程式開發,到發布完整的裝置韌體等不同的工作領域。

請參考我的夥伴 Julian 對此發布的文章

聯絡資訊:

John Lee
john@0xlab.org

Rename file extension to lower case

I have been too lazy to write this but today I was finally bothered enough. I don’t pipe it to sh inside awk because you may want to check what it will do before you actually do it.

#!/usr/bin/awk -f

# Usage: ls -1 | lowercase.awk | sh

match($0, /\.[[:upper:]]+$/) {
        if (RSTART == 1)
                next
        name = substr($0, 1, RSTART-1)
        ext = substr($0, RSTART, RLENGTH)
        printf("mv \"%s%s\" \"%s%s\"\n", name, ext, name, tolower(ext))
}

git-svn: fetch history from git clone

On http://trac.webkit.org/wiki/UsingGitWithWebKit there is a very useful trick:

If you want to be able to commit changes to the Subversion repository, or just want to check out branches that aren’t contained in WebKit.git, you will need track the Subversion repository. To do that, inform git-svn of the location of the WebKit SVN repository, and update the branch that git-svn uses to track the Subversion repository so that it will re-use the history that we’ve already cloned from git.webkit.org rather than trying to fetch it all from Subversion:

cd WebKit
git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit
git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master

This will add the following section to your .git/config:

[svn-remote "svn"]
url = http://svn.webkit.org/repository/webkit
fetch = trunk:refs/remotes/origin/master

You can then run the following command to have git-svn rebuild its metadata from your local repository, and to pull any recent commits from the WebKit SVN repository.

git svn fetch

So, let’s say you have a subversion repository:

svn://svn.openwrt.org/openwrt/trunk

and you also have a git clone of that repository:

git://nbd.name/openwrt.git

(Obviously these examples come from OpenWrt)
Now if you don’t want to fetch the whole history from svn, which will take forever, here is what you can do:

mkdir openwrt;cd openwrt;
git svn init --prefix=svn/ svn://svn.openwrt.org/openwrt/trunk
git remote add nbd.name git://nbd.name/openwrt.git
git fetch nbd.name
git config --replace svn-remote.svn.fetch trunk:refs/remotes/nbd.name/master
git svn fetch
  1. The git config line tells git-svn to use the history in nbd.name, so you won’t need to checkout the same commits from svn.
  2. Use --prefix=svn/ to separate nbd.name upstream and svn upstream.

Ubuntu as home media center

I’m really impressed. I’ve always wanted to put a small & nice HTPC in the living room. My old one is a full fledged desktop with 4 core, big ram, big hard drive, etc. Well at least it’s relatively ‘big’ when I bought it, but not anymore. The problem is — it looks really, really ugly in the living room.

And it’s running Windows XP.

Not that I’ve anything against Win XP. It’s stable enough, compatible with most of the stupid government and bank websites and applications, so I can use it to pay taxes, use web ATM, etc. It’s really useful. Mine is a legitimate copy, but I didn’t buy it. It’s actually a gift from Microsoft many, many years ago, in my previous life…

Anyway, the problem is that it’s hard to move to another PC. I’ve never be bothered enough to figure out how to use the ‘sysprep’ util, but start from scratch and update/install everything instead. It’s too much trouble, so it kinda stopped me from replacing it.

So, I was window shopping on the Internet and this Giada N20 caught my attention. It’s really small – 0.6 liter – and it comes with a remote control and HDMI. Video files could be decoded by Nvidia ION2. It’s exactly what I need. The real issue was that do I have to buy a Win 7 with it? People told me it’s really hard to use. Another thing was that the hardware is not particularly fast, so running win 7 might be too much for it. I figured I can try to install XP, since the manufacture said it’s fully supported.

I was so wrong.

I plugged in my ancient USB CDROM with the original XP CD in it. It booted into install screen and the process went just fine. I noticed that the reported hard drive capacity was incorrect, but I told myself it’s nothing and will be fixed as soon as I update the software.

No matter how hard I try (well not that hard, I just pressed the power button and F8 a few more times), it just cannot boot into XP. I thought it need some new drivers, but I couldn’t install the driver until I had a running XP, can I? Indeed it has an option saying that one may install 3rd party drivers during the install process, but it needs — floppy! Gosh, yeah I do have a floppy drive on my old machine, but who has a floppy with ‘USB’ interface nowadays? Guess I could just forget it.

In my desperation (I almost hit the shopping button for a win 7 copy), I figured maybe I can try to install Linux on it. It wouldn’t hurt. If it didn’t work I could put win 7 in it anyway. So I took out my trusted Ubuntu 11.04 USB flash and installed it.

After it booted for the first time, of course 3D didn’t work. I launched ‘Install additional driver’, it recommended Nvidia, I installed it, reboot… Yeah Ubuntu Unity interface was on, which means 3D was working. Then I enabled Mediabuntu, installed mplayer along with w64codecs, etc. Tried some high definition movies. There’s no audio output. Well, the display seemed to be fast enough, but it’s obviously running on the CPU so the codec was slowing things done. I clicked sound preference, picked HDMI 2nr (or something like that) then audio worked. mplayer picked vdpau as the video output, so I checked ‘mplayer -vc help’ and found some ffmpeg with vdpau support. I put these into ~/.mplayer/config, then 1024p movies played smoothly on my 52 inches LCD TV. Cool!

The remote control runs just like a keyboard input on this thing, and it can remotely power on/off this box. So I set the keyboard shortcut and set the default power button to ‘suspend’. Now I can resume and control it with a remote.

The last thing was to connect my Lavry DA-11 to it. It comes with USB interface and it’s standard USB audio, so there was no issue. I could switch sound output between HDMI and USB, depends on weather I want to hear the sound from the TV or my stereo set.

That’s about it. I must say it exceeds my expectation. Yeah the software is definitely there, but the hardest issue has always been the hardware. A media center relies on valid hardware drivers to work. Congrats to Ubuntu (and Linux) and Nvidia for going this far.