Twitter has been offline for the last 40minutes, and I really wanted to blip / tweet / ping the world that I had been enjoying some things…

Enjoyments

Have been enjoying the following without any guilt whatsoever :

- ‘religulous‘ doco last nite, very irreverend, very entertaining

- ‘Rereading Darwin‘ – still radical & relevant if we can grok eon sized time-spans

- web based interactive graphing calculator : www.abettercalculator.com

These are all awesome things, relying as they do on the advances that can be made when young people have freedom to think, to learn, have access to books and mentors and dont need to spend all their time either at war or gathering food or preparing shelter.

Decentralise!

So that was going to be the basis of my tweet, but that has been subverted into a rant on why Twitter, and Facebook, and Foursquare and LinkedIn and… are fundamentally wrong, whereas the Internet is fundamentally right.

Twitter and Facebook etc. [or the social networking needs they serve] should really be implemented as a distributed system. Each person or a cluster of people would have a local server ‘node’, and it would route messages to other nodes and so on.

For popular tweets or status updates, or checkins or wall postings there would be thousands of copies of that tweet sitting on servers all over the globe. This redundancy is a good thing, there would be less chance that a single Denial of Service attack or plain old human error or of a hardware failure or a mass event taking down the whole system. It could also mean we have more control of our own data – we should ‘own’ the posts we make, and the company we pay [ in advertising or in cash ] would provide the plumbing for the routing of messages and content.

This kind of distributed agent approach requires us to grow psychologically beyond the need for a ‘Central Authority’.. in fact I think that the technology will eventually take us forward to better forms of government, security and privacy, in that they are less centralised, more efficiently optimised to the local and the global environment – fundamentally they will offer more freedom and may require more responsibility.

Here are some impressions after using on a commercial iPhone app project :

My feedback after using Parse for real work for a few days on iOS is that there are a few things that Id like to see added :

• allow Pointer type cols [aka relations / object refs ] to be added via the web / DataBrowser
• allow double click on objectIDs in DataBrowser, for ctrl-C/V
• add csv export, json export [ imnso, these make me more likely to pay for Parse, not less ]
• search/filter, sort cols, in DataBrowser

As a workaround I had to do “inspect element” of the Data Browser web page to get the object id, then write some test code to create the object Pointer references.

Also Ive been pining for a detail button for each row – that would pull up a detail grid with [ colname | value ] so I can see all a rows attributes vertically at a glance [ if it can be left open while I select rows, all the better ]

Generally, the iOS Parse api is nice, and it just works… in fact a lot of this I had already implemented myself in js + Node.js + mongo, but a client wanted a Parse backend so that gave me an excuse to evaluate.

So my first impression is good, but as people use this for real work, youll need those little details to save time.

( btw, prefetch link/reference with includeKey is handy )

Lokenote is a very simple way to share notes based at a particular location… a kind of geo PostIt note taker.

• Heres an example of a Lokenote.
• Go here for the Lokenote app in iTunes.

There are great apps out there such as Gowalla, FourSquare, Yelp but I felt the need for something ultra simple with low overhead where you dont need to register or signup – just drop a note where you are, and leave it for other people to find.

I still see notices pasted up on supermarkets and on lampposts for Missing Cat,  Part-time Work,  Flatmate Needed or Garage Sale.. so I think there is a need for this kind of utility belt app.  I specifically wanted a tool to jot down nice graffiti snippets from around St Kilda where I live, and mention handy places a non-local might not know about such as the well hidden laundry shop.  Id like to give a virtual nod to some of the superb out of the way food places that exist in my version of Melbourne.

Next

In developing Lokenote I took the ‘Fire!… Ready? Aim‘ approach – I had a rough idea, started as simply as I could, implemented feverishly and only added things I thought were absolutely essential.

Not being able to entertain any extra features had the perverse effect of generating many more ideas.. but these had a more organic character growing out of reality.  Beware ideas that have clean academic edges, they tend to not fit the world.

The process of building Lokenote gives me a a furtive and voyeuristic sense of the kind of realtime app which I think is just around the next bend.

We can safely assume Next apps are :

• mobile/web hybrids with touch UI
• reactive in realtime, via flowing data feed
• location aware, fulltext searchable
• online/offline robust
• built on graph style data models

But what might they actually do ?

• live auction or product sales [ the last 15mins of an eBay auction without the 6 day lead-up ]
• convergence of blog, web page and chatroom with live comment feeds
• realtime automated sentiment, trend summary
• flexible links between any kinds of data
• scrolling realtime chat, tethered to a location, keyword/topic, group or event

Feedback

Feedback is the best word I have to describe the qualitative difference of realtime apps.

An example – rather than go to a conference and wait for surveys to come in from attendees and adapt in time for the next event, the feedback loop is immediate enough to customise as it progresses.  This already happens, with some presenters saying ‘tweet me if you want more or less on this topic’ – a tweet is more anonymous and less impolite than interrupting the speaker, and feedback is current.

We might see more prices that are changing moment to moment, or other micro-optimisations –  the cost of a flight might be offered within a range, and be fixed only once the aircraft leaves the gate.

Realtime Dating?

Another example of a realtime next app is for dating.  I envision this as a kind of randomised, localised topic-chat :

• nominate a topic and post a comment, or join an active topic that looks interesting
• chat away anonymously for a while
• notice someone interesting, share your profile
• get a nudge back or an invite for a one-on-one chat
• if things progress, decide to meetup at a cafe on neutral ground

Most dating apps use the profile photo as the initial filter.. but Im not so sure that is the initial filter in Life.. sometimes people with unremarkable looks win you over and in fact become more attractive over time as conversation unveils their personality.  So conversation as the initial filter might actually work.

The very same app could be a great way to generate ideas in business or science or political activism… it just seems the old chat room needs to be upgraded for the realtime web, so that it resides next to all the other things I do on the web.  I might want to attach a web page or doc or graphic or photo or video to my realtime comment.  Parties might agree to go private with some comments.  You might want to limit the audience to a group or post anonymously then go back on-record.

Lets Build

We have all the plumbing to do this feasibly – technologies such as nginx, node.js, Mongo, Couch, Riak, Redis, Web Sockets, JSON, HTML5 are really at the point of becoming the normal way to write dynamic data-driven responsive web/mobile apps.

Its about taking some risk to walk over the local maxima and build these things that will make life simpler, leaving more time for people to enjoy the roses.

Well, those are my thoughts for now… enjoy, gord.

Here is a minor update I did a while ago that shows what happens when a second circle spins around the point on the outside of the first circle…

[click the pic or here to view the animation ]

This might be a nice lead into Fourier Series…

Some impressions / notes -

• github.com is superb!  Radically better/simpler/easier/nicer than sourceforge or google code
• I lurrve code snippets, aka GISTs hosted on github. Sane blogs support inline gists [ not wordpress.com, yet ]
• Found a clear readable tutorial on setting up a remote git repo (with git and gitosis) for private or public use
• Github is so nice, its really tempting to just pay them money for some private repos rather than step thru the above
• A handy git meta- cheat sheet here

So that we can reuse the simple serialq code from the previous blog post, I have tidied things up and packaged into an NPM module.Apparently its now installable using “npm install serialq”.

Creating the module was a breeze -

• within your directory run ‘npm install’ and fill in the questions
• npm adduser, npm publish
• test with npm install

After publishing, the module magically shows up on the extremely handy npm.mape.me module search site, under keyword ‘serial’.  See isaacs article ‘How to Module‘ for overview.

Code

The code is a bit simpler to read, as you can see its a very short implementation :

exports.SerialQueue = function()
{
    var sq = 
    {
        funcs : [],
        next : function()
        {
            var Q = this;
            var f = Q.funcs.shift();
            if (f)
                f(function() {Q.next();});    
        },
        add : function(f)
        {
            this.funcs.push(f);
        },
        run : function()
        {
            this.next();
        }
    };
    return sq;
}

Usage :

    var Q = SerialQueue();
    Q.add(fn_first);
    Q.add(fn_second);
    Q.run();</code>

[ For a more readable version of the code snippets above, see this github Gist. Would be nice if wordpress.com supported Gists, Posterous do..ahh maybe time to move my blog. ]

Thoughts

I found this module handy for serializing access to a mysql database. Breaking out this boilerplate made the rest of the code clearer.  Code is up on github

Surprisingly Javascript + Node.js is a real workhorse.  I actually prefer it to Perl/PHP and even Ruby/Python for data plumbing tasks.  You have hashmaps and regex handling built in, garbage collection, and a superb general purpose data format in JSON.  Perhaps Javascript is the hundred-year-language?

Be aware this is ‘cooperative’ sequencing.. each function gets passed a done or next argument, and will have to invoke that to signal completion [ causing the next function to be run ].

The whole crux of Node.js is that everything is done async by default – you fire off something now and get a callback at some later time.  Its a beautiful paradigm and means that you can get great performance, because it fits so closely with the underlying operating system calls [ libevent, completion ports, sockets etc. ]

However, there are times you do need to enforce serial processing… for example checking for valid user/password must return a result before getting sensitive data and displaying it on the web page.

There are now sophisticated serial modules for Node, which Id recommend you look at for real work.  For example Conductor can mark sequential dependencies and will allow the most async processing to happen, while honoring those sequential constraints – the best of both worlds.  Another nice approach is this fork() primitive via stackoverflow.com.

Lets have a look at the simplest case, to see whats under the hood…

Demo Code

I made a test program to compare sync versus async,  This illustrates a very direct approach for serial processing using a queue of work functions. Code on github, here : async_vs_serial.js

For those of us new to Node, its useful to look at the simplest serial implementation.

There are three slow functions : do_aaa(), do_bbb(), do_ccc().  These can be run serially or in parallel.

In the serial implementation I pop them off the front of the work queue and run each in turn.  For this to work, each work function calls next when its callback completes.

Sample Output

    Default run [ Async ]

        > time ./async_vs_serial.js async

        RUNNING ASYNC
        >aaa
        >bbb
        >ccc
        <bbb took 50ms
        <ccc took 120ms
        <aaa took 220ms

        real    0m0.325s
        user    0m0.090s
        sys 0m0.012s

    Serial run

        > time ./async_vs_serial.js serial

        RUNNING SERIAL
        >aaa
        <aaa took 220ms
        >bbb
        <bbb took 51ms
        >ccc
        <ccc took 120ms

        real    0m0.479s
        user    0m0.075s
        sys 0m0.011s

The async functions are nicely interleaved, saving wall time.

Reality is Async by default

I think its best to think async by default… after all the CPU can do a huge amount of work while its waiting for IO to complete.  Its no surprise were seeing a whole raft of realtime web apps being powered by Node.js.

For background, see Ryan Dahls talk notes on ‘doing IO differently‘

Some handy links before I forget  :

• howto.no.de articles – especially Part I, II and III of ‘Learning Javascript with Object Graphs’
• Joyeur blogs – people working for Joyent on Node / DTrace / Solaris
• no.de Joyent Node hosting [ built atop - 'open' solaris, ZFS, DTrace ]
• NodeJitsu blog
• npm.mape.me – searchable Node.js Modules list
• connect-it guide – web framework with chained middleware layers
• express guide – article on express web framework for Node
• eventserver – Tom Lee’s internet tee piping for notifications

In other news.. Im hacking over ssh via a long thin pipe to my linode server – using a very erratic mobile broadband connection, arrgh!

Cant wait for ADSL to _finally_ be connected here, so I can watch Bryan Cantrill talk about Cloud Analytics :]

As an aside.. why Javascript?  Consider

• Javascript is a totally distinct language from Java
• Javascript deserves its bum rap.. to mis-quote Dame Judi Dench, its bad parts are “arse-clench-ingly” bad :]
• The good parts of Javascript feel very nice, like a modern lisp inspired language, fairly concise, many valid idioms
• V8 js engine is fast
• Javascript callback mechanism fits async event IO really well
• Node.js embodies the above bullet point into a fine server development environment
• JSON, the Javascript native data format, is all the good things of XML with none of the bad
• feels like a unix-like web-plumbing philosophy
• can keep the same language syntax hat on when writing front end web apps and back end servers

• HN style forum of quant posts – quant.ly
• High Frequency Trading articles are here
• Herb Sutters Effective Concurrency articles from Dr Dobbs Journal are here
• Herb Sutters Blog – Sutters Mill
• Fun talk on C++ – vid here [If the goal of C++ is 'high-performance-abstraction' has it succeeded? ]

Rhetorical question – given that only three people on the planet truly understand the subtleties of C++ … is this a language anyone should be using to build reliable software?    C++Ox is nearing completion, so the language continues to grow in size.

You can view it using Safari or Chrome browsers.  For iPhone, open in Safari and save to Home Screen, then you click the icon to load [for some reason fails to load directly, but works fine launched from Home Screen ]

For the path checking.. at first I did an exhaustive depth first search of all empty squares, but this exploded combinatorially.. so I used memoizing to reduce the repeat subpaths.. and then added a heuristic to find straight paths if they exist.  I probably should have looked up a decent path finding algorithm.  Sometimes its good to do things the blunt way yourself, so that you understand the problem and can then enjoy the beauty of a more elegant approach.

Hope you like the pictographs.. did them myself :]

Backstory

Lost in the dense Cambodian jungle you see what looks like an outcropping of ancient ruins.  Could that plynth be the entrance of an alien spacecraft which has lain dormant for thousands of years?  You wipe away the dust showing an irregular pattern of grooves.. you accidentally spill some water on the  console.

You hear a hiss followed by a gentle deep hum, and the panel under your palm begins to glow.   You must quickly match the glyphs to enter the Xeno’ ship…