Wednesday, July 30, 2014

Tail Recursion

One of the cool things in Erlang is efficient tail recursion. It means that when a function calls itself as its last instruction, Erlang re-uses that stack frame. That probably doesn't clarify it much, so let's look at an example.

This Java code

public class Recursor {
    public static void main(String[] args) {
        System.out.println("Total: " + countdown(10));
    }

    private static int countdown(int x) {
        return countdown(x, 0);
    }

    private static int countdown(int x, int total) {
        if (0 == x) {
            return 1/0;  // generate exception
        }
        else {
            // recurse with decreased countdown and increased total
            return countdown(x - 1, total + 1);
        }
    }
}
generates this stacktrace
$ java Recursor
Exception in thread "main" java.lang.ArithmeticException: / by zero
 at Recursor.countdown(Recursor.java:12)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:15)
 at Recursor.countdown(Recursor.java:7)
 at Recursor.main(Recursor.java:3)

If the countdown had started at 100, you'd have a hundred lines in the stacktrace.

The equivalent code in Erlang

-module(recurse).

-export([countdown/0]).

countdown() ->
    countdown(10, 0).

countdown(0, Total) ->
    Total/0;  %% generate error
countdown(Val, Total) ->
    %% recurse with decreased countdown and increased total
    countdown(Val - 1, Total + 1).

generates this stacktrace

35> catch recurse:countdown().     
{'EXIT',{badarith,[{recurse,countdown,2,
                            [{file,"recurse.erl"},{line,9}]},
                   {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,573}]},
                   {erl_eval,expr,5,[{file,"erl_eval.erl"},{line,357}]},
                   {shell,exprs,7,[{file,"shell.erl"},{line,674}]},
                   {shell,eval_exprs,7,[{file,"shell.erl"},{line,629}]},
                   {shell,eval_loop,3,[{file,"shell.erl"},{line,614}]}]}}

Even though the function has called itself recursively ten times, it's only one line of the stacktrace. (I've tested it up to 100 million.)

In fact, this is true of any tail call, even to different functions. This code generates the same stacktrace

countdown() ->
    countdown(100, 0).

countdown(0, Total) ->
    Total/0;
countdown(Val, Total) ->
    bounce_one(Val - 1, Total + 1).

bounce_one(Val, Total) ->
    bounce_two(Val, Total + 1).

bounce_two(Val, Total) ->
    countdown(Val, Total + 1).

Surprisingly, even non-tail recursion in Erlang is done fairly efficiently. This version of the code, where the last instruction is addition

countdown() ->
    countdown(10).

countdown(0) ->
    1/0;
countdown(Val) ->
    1 + countdown(Val - 1).

generates this stacktrace

> catch recurse:countdown().
{'EXIT',{badarith,[{recurse,countdown,1,
                            [{file,"recurse.erl"},{line,9}]},
                   {recurse,countdown,1,[{file,"recurse.erl"},{line,11}]},
                   {recurse,countdown,1,[{file,"recurse.erl"},{line,11}]},
                   {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,573}]},
                   {erl_eval,expr,5,[{file,"erl_eval.erl"},{line,357}]},
                   {shell,exprs,7,[{file,"shell.erl"},{line,674}]},
                   {shell,eval_exprs,7,[{file,"shell.erl"},{line,629}]},
                   {shell,eval_loop,3,[{file,"shell.erl"},{line,614}]}]}}

Even there, there are only three stack frames for that recursive function. (It's the same even with a recursion depth of 100.) I'll have to find a real Erlang expert to explain that.

The reason this is important is that it's the basis for the Actor model in Erlang. Actors are long-running processes that fundamentally look something like

loop(State) ->
    receive Message ->
        NewState = handle_message(Message, State),
        loop(NewState)
    end.

They wait for incoming messages, do something with them, and recurse with their updated state. Yes, you could do that in Java with a while loop, but doing it with a recursive function has two benefits. The first is just cleanliness: When you call a function, it just gets the values that are passed to it. In a while loop, you have to worry about the state of any variables that are visible from inside it.

The big win is that this is what lets Erlang do hot code loading. All of the process state gets passed as a parameter, so its data is separate from its code. When loop calls itself, it can invoke the new version of its code. Because Java classes combine code and data, you can't update the processing logic without wiping out the application state.

Wednesday, July 9, 2014

HTTP request using caching

Make a normal request for the resource
$ curl -si 'http://localhost:8080/myapp/tmp.json'
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"43-1404927802000"
Last-Modified: Wed, 09 Jul 2014 17:43:22 GMT
Cache-Control: max-age=315619200
Expires: Tue, 09 Jul 2024 18:04:57 GMT
Content-Type: application/json
Content-Length: 43
Date: Wed, 09 Jul 2014 18:04:57 GMT

{
 "first": "Colin"
 "last": "MacDonald"
}
Grab the ETag from the response and send it in the 'If-none-match' header
$ curl -si -H 'If-none-match: W/"43-1404927802000"' 'http://localhost:8080/myapp/tmp.json'
HTTP/1.1 304 Not Modified
Server: Apache-Coyote/1.1
ETag: W/"43-1404927802000"
Date: Wed, 09 Jul 2014 18:05:10 GMT

Now update the timestamp on the file.
$ touch /usr/local/Cellar/tomcat/7.0.53/libexec/webapps/myapp/tmp.json
Then request it.
$ curl -si -H 'If-none-match: W/"43-1404927802000"' 'http://localhost:8080/myapp/tmp.json'
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"43-1404929189000"
Last-Modified: Wed, 09 Jul 2014 18:06:29 GMT
Cache-Control: max-age=315619200
Expires: Tue, 09 Jul 2024 18:06:36 GMT
Content-Type: application/json
Content-Length: 43
Date: Wed, 09 Jul 2014 18:06:36 GMT

{
 "first": "Colin"
 "last": "MacDonald"
}
You get the full response, with the updated ETag header.

Friday, June 27, 2014

Why I Hate Java, pt. 1

To get the body of an HttpResponse as a string, you have to do this:

HttpEntity entity = response.getEntity();
InputStream content = entity.getContent();
InputStreamReader contentReader = new InputStreamReader(content);
BufferedReader reader = new BufferedReader(contentReader);
StringBuilder builder = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
    builder.append(line);
}

Am I the first person on the internet to ever do this? Why on earth isn't there a convenience method like getEntityText() or something?

Thursday, August 22, 2013

ssh to ipv6 link-local host

What I wanted to be able to do was run my Raspberry Pi as a headless server, and just plug an ethernet cable into it from my laptop and ssh in. Is that so much to ask? I expected it to be automatic with ipv6, since it has automatic host discovery built in. No so much.
  1. ipv6 is not enabled on a Raspberry Pi by default. Edit /etc/modules.conf and add a line for "ipv6". Reboot.
  2. You need to know what address to connect to. ip neigh will show you your neighboring IPs, but it doesn't know about the link-local hosts; you need to ping them first. But you need their addresses to ping them, don't you? The secret is to ping the magic link-local address, ff02::1. Then ip neigh will tell you what you need to know.
    $ ping6 -c 1 -I eth0 ff02::1
    PING ff02::1(ff02::1) from fe80::226:2dff:fef9:3f85 eth0: 56 data bytes
    64 bytes from fe80::226:2dff:fef9:3f85: icmp_seq=1 ttl=64 time=0.050 ms
    
    --- ff02::1 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.050/0.050/0.050/0.000 ms
    $ ip -6 neigh
    fe80::ba27:ebff:feb6:6647 dev eth0 lladdr b8:27:eb:b6:66:47 DELAY
    
  3. Ok, I've got the address, but ssh fails with a cryptic "Invalid argument" message.
    $ ssh -6 pi@fe80::ba27:ebff:feb6:6647
    ssh: connect to host fe80::ba27:ebff:feb6:6647 port 22: Invalid argument
    
    This is because ssh (actually, the kernel) doesn't know which interface to talk to. Adding a route for the link-local host doesn't seem to work:
    $ sudo route -A inet6 add fe80::ba27:ebff:feb6:6647/128 dev eth0
    $ route -A inet6 | grep eth0
    fe80::ba27:ebff:feb6:6647/128  ::                         UH   1   0     0 eth0
    fe80::/64                      ::                         U    256 0     0 eth0
    ff00::/8                       ::                         U    256 0     0 eth0
    $ ssh -6 pi@fe80::ba27:ebff:feb6:6647 
    ssh: connect to host fe80::ba27:ebff:feb6:6647 port 22: Invalid argument
    
    So instead, I specified the interface in the ssh hostname (by adding it after a '%'). Here's what that looks like:
    $ ssh -6 pi@fe80::ba27:ebff:feb6:6647%eth0
    pi@fe80::ba27:ebff:feb6:6647%eth0's password: 
    
    scp has a similar problem, and it also uses colons to separate hostname and filename, so you have to put the colon-laden ipv6 address inside square brackets
    $ scp -6 my_file.txt pi@fe80::ba27:ebff:feb6:6647%eth0:
    ssh: Could not resolve hostname fe80: Success
    lost connection
    $ scp -6 my_file.txt pi@'[fe80::ba27:ebff:feb6:6647%eth0]':
    pi@fe80::ba27:ebff:feb6:6647%eth0's password: 
    
I ended up writing a shell script to do all that setup for me:
#!/bin/bash

user=$1
test -n "$user" || user=pi

iface=eth0

# Find the ipv6 address for our link-local host
linkhost=`ip -6 neigh | cut -d ' ' -f 1`
if test -z "$linkhost" ; then
    ping6 -c 1 -I $iface ff02::1 >& /dev/null
    linkhost=`ip -6 neigh | cut -d ' ' -f 1`
fi
echo "Found link local host $linkhost"

echo "ssh -6 -X $user@$linkhost%$iface"
ssh -6 -X $user@$linkhost%$iface

Friday, May 10, 2013

Wireshark Config

Wireshark is a GUI tool, so it runs as your normal user, but it needs to be able to capture packets, which is normally a superuser thing. There is a facility for giving a normal user limited permissions to capture packets, but that's not something you want enabled by default. All good and reasonable, but it has the unfortunate consequence that you can't actually do anything with Wireshark out of the box. Fortunately, they make it pretty easy to enable your permissions. You can just run this, and it brings up an ncurses UI to let you enable non-root packet capture.
$ sudo dpkg-reconfigure wireshark-common
It sets up a wireshark group and gives it permission to capture packets (using the dumpcap utility). Then you need to add yourself to the wireshark group.
sudo usermod -a -G wireshark myuser
You'll have to log out and back in for your user session to have the right permissions.

Wednesday, May 2, 2012

JSON pretty-printing

Today's magic invocation is for pretty-printing JSON strings. If I hit a REST service with my browser, I get back a wad of JSON, which is great, but a bit hard to read. In Chrome (at least), I can pop open the Javascript console and run this:
JSON.stringify(JSON.parse(document.body.children[0].textContent), null, 4)
The JSON is actually displayed by Chrome as text in a <pre> block, which is the only element on the page (child 0 of body). So I have to grab that text, parse it into JSON, and then pretty-print it using stringify (with a 4-space indent). Thanks to Stackoverflow for pointing me in the right direction.

Monday, April 16, 2012

vim config w/ pathogen

From article on Vim plugin management

What I actually did was install pathogen as per https://github.com/tpope/vim-pathogen (more or less)

mkdir -p ~/.vim/autoload ~/.vim/bundle; curl -so ~/.vim/autoload/pathogen.vim https://raw.github.com/tpope/vim-pathogen/HEAD/autoload/pathogen.vim
echo -e "\n\ncall pathogen#infect()\n\n" >> ~/.vimrc

Then manually install just the few plugins I really wanted.

cd .vim/bundle/
git clone git://github.com/pangloss/vim-javascript.git
git clone git://github.com/timcharper/textile.vim.git
git clone git://github.com/tpope/vim-markdown.git
git clone git://github.com/nono/vim-handlebars.git
git clone git://github.com/kchmck/vim-coffee-script.git
mkdir jQuery
curl -o jQuery/jquery.vim http://www.vim.org/scripts/download_script.php?src_id=15752

Jekyll Pages List

Getting a list of pages in a Jekyll site is easy, but paring that down to just the top-level files and the index pages is tricky. {% for p...