tiyuti web store tech demo – Classic Watch Men

Team tiyuti – gord and paul – have been busy building a new web shop technology, using the same deep indexing tech we used to search 40k Melbourne restaurant dishes ..

demo shop here : “Classic Watch Men” : https://demo.tiyuti.com

Built with hand-crafted javascript : node.js and postgresql under the hood.

[ Looking for people who have a few thousand products to sell, to showcase our tech .. emailing support@tiyuti.com will reach the founders :p ]

#ecommerce not #shopify not #woocommerce #webstore #webshop #indexing #nodejs

node.js perf followup ..

From the previous article, a few people mention 10 writes per second is pretty damn slow – and they have a point.

So I tried pushing things to the edge of where performance starts to degrade …

I ran a whole lot of writer processes and some reader processes .. and I see a steady 120/sec WRITES while still having good response to reads / low latency.

Write pressure begins to degrade performance

Beyond that, if I push it up to 160 to 200 WR/sec I see read performance degrade ..ie. instead of latency being <80ms .. they blow out to 250 .. 500ms .. then go up dramatically as the system has to queue reads and writes, buffers them and tries to catch up.

So I guess the best conclusion I can draw is that out of the box performance node.js+postgresql under simulated real-world usage, gives you around 120 writes per second while not degrading performance.

Thats not too bad .. 100 writes/sec would be a lot of users hitting your average web site .. but admittedly there are a lot of optimizations one could make.. this was more about how far you can get without doing any clever performance tweaks [ redis, message queues, server sharding yadda yadda ] ..

Well, the MON idea was really about getting visibility into host latency .. once you do that you then have actual data, that may or may not match your intuition.

tldr : I pushed writes to 120 wr/sec before seeing degraded performance

PhoTree : search 10k Books or 30k dishes

At tiyuti.com, we’re testing out general performance and usability of PhoTree : share your list ! .. so imported some realworld data to try things out :

• Something to read ? search 10k books here : Big Famous Book List

• Something for Lunch Dinner Date ? search 30k dishes : Melbourne Food Finder

Performance seems pretty good – both pages scored around 95 when tested on pagespeed.web.dev

Content feels like its loaded after second or two .. not too shabby for a fully javascript SPA [ single page app ] which gets all its data as json then generates the page elements… and there are plenty of places to reduce latency. One thing we are glad we did is gzip compress, which helps reduce a large json data payload footprint.

mon + pho : aka How fast is node.js + postgres ?

Been busy building PHO aka “PhoTree : share your list” a kind of photo list link tree. The idea is simple, you make a list of your cool stuff and share it, as easily as possible – no login, no email, no signup process – just make a new list and start adding items with pics to it.. within seconds.

We wanted all text to be automatically indexed so you can search anything – ie. you can search for a book by author, title, isbn topic / whatever. We imported a few thousand books as a test – check out the “Big Famous Book List” if you’re looking for something to read 🙂

We [ me and my son Paul, on holiday from uni .. aka ‘Team Tiyuti’ ] started with the web json api first and built PHO over the past week and a half of hacking javascript and css. Instead of the usual register/email/login/password guff, we went for a secret key needed to edit a given page – so we don’t harvest peoples information and you can make a new page instantly.

We built it using node.js and postgresql .. and I was asked a few weeks back by a colleague “is node.js fast? ” .. my impression is its pretty damn fast, for a scripting language. Later I thought well, it would be nice to actually listen in on all the web and json api endpoints, and get latency stats … ie. evidence rather than hearsay / intuition.

So.. I hacked up MON – a latency monitor for node apps. MON wraps your node.js http handler, looks at the request coming in, and the response going out and measures the time delay ‘dt’ or ‘latency’. It keeps track of latency and counts for each api endpoint url that comes through and serves up a summary page.

MON latency and counter stats for each endpoint

I couldn’t resist adding the blinken lights that flash green when a request comes in : )

To get an idea of the performance we can expect from a web app built with node.js and postgresql database on a common or garden VPS server with SSD disk .. we need to put the system under some simulated user load. So I wrote reader and writer test scripts to make pages and items and fill them with random words, and read them back in any order etc .. the above tests shows a fairly heavy user load of ~10 writes per second and 20 reads per second .. and even then the median latency is well under 50ms.

I did notice a few outliers at around half a second .. which may be due to warming up the DB cache, or possibly garbage collection .. which I will look at more in future. The human eye takes around 200ms to blink, and if the roundtrip + page rendering is in the order of a second, most users will experience that as very responsive.

My gut feel is that node.js itself is not doing a lot of work, basically acting as a router, sending requests for files to linux filesystem and sql requests to the database .. so what we are seeing here is basically excellent performance by postgresql, linux and modern SSD backed hosting environments.

If you enjoyed this read, you might also want to have a look at another very terse article on a similar topic : “web search 30 million rows sub-second, cheaply”

enjoy : gord at tiyuti dot com

ps. you can hire us to build your new new – but fast – thing :p

A-Levels from Home – our plan

Here are my thoughts on sitting the British “A-Levels” in Australia as a possible path to University Entrance for Home Educated students.

Disclaimer :  This is a guide only – our current plan which we have not yet verified/ completed – so please check for yourself before making any decisions.

Questions :

• What are A-Levels ?
• Why do them ?
• How to do them ?
• Costs / Logistics ?
• Does it really work ?

In a nutshell, the A-levels are the UK / British approximate equivalent of the Australian VCE/HSC/ATAR, or the American CollegeBoard/AP exams, or the IB International Baccalaureate.  That is to say, they are the highest secondary school qualification, and are used to gain entrance to University and satisfy tertiary prerequisites.

A-levels are sat in many international schools worldwide, and are quite well known by universities in many countries, so the qualification is reasonably well regarded and can usually be compared/converted to the local equivalent.  The exams are standalone and not based on coursework, they are marked by an independent body and have few formal prerequisites or formal age requirements.   There are plenty of course materials, books and past exam questions available, and A levels cover a wide range of subjects in depth.

In Australia, the BritishCouncil in Sydney run the exams in May/Jun and Oct/Nov, and some in Jan.  There are various exam “boards” which set the exams and mark them, and BritishCouncil offer the option of EdExcel or CIE ‘flavor’ exams.

In our case, DS15 – code for ‘my 15 year old Darling Son’ – has worked a couple of years ahead in Math, using materials from KhanAcademy.org, AoPS.com and amt.edu.au.    Doing some fairly rigorous academic tests in Math next year seems a good fit, in preparation for University the following year.   If we could have done VCE methods, ‘spesh’ and physics next year that would have worked out fine – but we could not find a school that would allow us this flexibility.

In the UK it is normal to sit 3 A-levels, so our plan for 2019 is roughly :

• “Maths” in May/Jun
• “Further Maths” in Oct/Nov
• “Physics” in Oct/Nov

I think this gives us a reasonable chance of getting into a good science/math program at an Australian university, but they may insist he also sit an entrance STAT exam to show evidence of English / Reasoning / Writing skills at university level.

The A level exams are not super cheap – each test is around $300 AUD, with usually 2 or 3 tests per subject on different days, so around $2400 in our case.  There is also the issue of travel and accommodation to sit the tests in Sydney and lesser costs for books and materials.  There are some online distance ed providers for A levels, but in our case we prefer to self study from the books.

If things go well, we will report back on our success and this pathway might prove useful for other home educated students.  If things don’t go well, we can review what went wrong, and DS will still have time to find other pathways to get into Uni.   I admit this is a pretty “geeky” approach with so much emphasis on Maths, but it suits us well and DS does have wider interests – he plays the violin, reads voraciously, writes, does computer art and programming, loves cooking etc.   Even with good marks in A-levels a 16yo student will most likely need the University Dean of Schools approval, so we shall see what they say about our best-of-british quals !

Some links :

• British Council Australia IGCSE and A-Levels
• Pearson Edexcel A-Levels
• CambridgeInternational / CIE A-Levels

moosh.io – instant web wallet point of sale

Initial beta release of my instant web wallet for Bitcoin Cash : moosh.io

I talked to a few people about getting started with BCH, but they had issues setting up a wallet to accept payments.

The first time you visit moosh.io, it creates a random wallet address for BCH, so you can receive funds and send them on.

The other problem is how to accept payments as a small business – a cafe, restaurant or food stall.  Basically you need a simple way to know which order/product/sale matches up to which payment – who has paid for what.  I achieve this by having a temporary BCH address to accept the payment for a given order, and then forward it on to the main vendors wallet as soon as payment is detected.

All the keys are kept on the client side, in localStorage on the local device, and transactions signed within the browser.

Ill be extending this to look nice and work well on mobile devices – now that money is purely digital, all those heavy point-of-sale hardware can be replaced by a nice web app running on your phone.

Multiply with Box Method – “number-sense”

I was discussing with other parents online, about the merits of various school Math multiplication approaches.

In the “Japanese Line method” you draw lines for each digit, and then count the intersections and add them up.

In the “Box-Method” you draw areas for each sub-product, and in “Longhand Multiplication” you just use the square grid to keep track in a table, without use of a diagram.

My main issue with the Line Method is that for problems like 97*98, you’ll be drawing a lot of lines .. its a lot of work even for 64×34 :

I also like the idea that Box Method starts with a physical / visual intuition – you can start children counting rows and columns, then progress to counting areas such as “How many 1ft square tiles do I need to tile the 3’x7′ kitchen floor ?”

Then you can discuss the distributive law a(b+c) and (a+b)(c+d) and explain the method behind longhand multiplication .. and then go on and show some handy tricks, such as using negative numbers to make less work, vis :

The above example, which does the same problem in 2 different ways, shows the kind of approach Jo Boaler recommends, called “Number Sense” – where students are encouraged to find their own way of doing a math problem.  The idea is that there are many valid ways to do it, and struggling to find your own way is a good thing [ even if you fail ], and students tend to engage more, and certainly recall more when they have actively discussed with other students.

The same Box Method diagram can lead onward naturally to algebra and quadratics.. and even the beginnings of calculus 🙂

Blockchain simulator in node.js

For experimenting with some ideas on improving blockchain scaling :

https://github.com/justgord/blksimjs

Currently simulates the growth of a crypto-currency blockchain ( similar to Bitcoin ), loops thru a cycle of :

• spend – pick addresses at random and spend to another address
• mine – put waiting transactions into the block, mine the cryptographic hash, add to chain

Features / Limitations :

• uses single sha256 bit hash for ids
• uses an easy Proof-of-work [ 256 tries on average to get a block hashs with ’00’ leading bytes ]
• uses node.js byte buffers for transactions and blocks
• runs around 7000 tps on i5 laptop

Motivation

I wanted to simulate the growth of a blockchain with unspent transactions spread somewhat sparsely at the early older parts of the blockchain, and more dense at the top of the blockchain [as more recent transactions havent had time to be spent yet ].

The reason for this is to test the feasibility of reducing the size of the data needed to bootstrap a new node. eg. in Bitcoin the whole dataset is :

• around 150GB of transactions [ 250Mn txns ]
• utxo of around 2GB [ ~50Mn txns ]
• so unspent ‘utxo’ set is around 20% of transactions

Bring UTXO set forward

We can use much less data [5x smaller ] when spinning up a new node, by bringing utxo set forward to nearer the front of the chain. The sim gathers old utxos and injects them into the blockchain in baches of ids, so they are stamped into the block at block creation.

These ‘utxo catchup sections’ are read when starting a new processing node – ie. it only needs a provable list of utxos, not the complete history of all spent transactions.

skip links [ todo ]

Using block extension areas, we can also include skip links to blocks much earlier in the chain – the process of walking back thru the links of transactions inputs and outputs all the way back to the initial ‘genesis’ block is like walking a DAG tree, as un-needed areas of the blockchain are skipped over.  These skip links are validated at block creation time by other nodes.

This is useful for clients which want to traverse the chain to make better proof of validity that SPV, and for nodes that use utxo bring forward above, so they can trace PoW to an arbitrary level back to the genesis block.

Bitcoin – dynamic block size

 

 

Bitcoin is a work of genius .. it is a world-changing technology revolution. But being the first incarnation of a new new technology, there are some things that can be improved upon :

• time to next transaction
• time for transactions to clear [ unpredictable processing times ]
• transaction fees
• energy efficiency

For now, lets just look at one issue – Currently in bitcoin, transactions are throttled by money supply/proof-of-work, and this is a bottleneck on transaction throughput. The fact is they don’t need to be, they can be decoupled.

Some eye opening stats :

• mempool sizes from 2MB to 30MB [ transactions waiting for a 1MB block ]
• We are “Hitting the wall“
• Bitcoin Transaction Fees vs Blocksize
• Transaction processing time vs block size :

[ last 3 images from woobull.com ]

But.. we can actually have our cake and eat it, when it comes to block size. Its not that hard a technical problem to solve :

Dynamic block size [ decoupled block size from money supply ]

The mining reward should be given purely for PoW / solving the next block hash / making the next “time-stamp” – it should not be a throttle on transaction volume. It is a throttle on transaction volume currently, because the block size is fixed per solve.

Currently we have blocks that are 15% full, and blocks that are 99% full – the block size is not the issue, the issue is the block size is fixed.

AUS startups need to Recycle talent, more.

I just read some good insights on the Aussie startup scene downunder, from Airtree VC partner John Henderson  :

http://www.businessinsider.com.au/a-top-executive-explains-whats-missing-from-australias-tech-startup-scene-2017-2

My own take on this is  –  RECYCLE talent, more.

We somehow need to reach a kind of critical mass of local success, so we have a sustainable “pyramid” of biz / tech / investment talent.

In Silicon Valley and even Berlin, as soon as startup X tanks [or exits], the people who worked there – devs, sysops, seo marketers, finance, bizdev, CXOs, managers, investors, scientists – move on to other startups and so are “recycled” back into the pyramid of special startup knowledge, experience and talent.

In startups, this is a vastly more important process than in normal business, because the tech and biz approach of early high-growth startups is fairly unique, and because its well known that all the economic benefit comes from the few successes – so there will be a lot of failures, and it would be really costly to waste all that experience / investment in time.   If people with startup experience move on to ‘normal’ business environments, instead of being able to move into another startup, its a massive loss.

One side effect of not having this self-replenishing Pyramid, is knowing both sides – techs who know biz, and entrepreneurs that know some code, founders who can wear both hats.   Aussie founders often have a view of tech as purely a cost center –  a pain point to outsource, a necessary evil – rather they should see it as core business, an area to innovate, to generate business ideas, a channel to the customer, a means to delight users, and a way to project power at scale.  I really think this is holding us back.

To get to this Pyramid, one thing we have to do is find more efficient ways to recycle talent – just get people to flow into the next thing, instead of a deep dive of depression and naval gazing about how we did the wrong thing and that’s why it tanked.  Do the painful postmortem blog post, learn and move on.. heal while your working on the next thing.  Recycle what you can – code as open source, your team into an acqui-hire or other startups via intros to people you know, even competitors.  Slava Akhmechet, founder of RethinkDB,  was totally classy in the way he did this.

There’s a scene in one of those bad Vin Diesel movies, where Vins getting stared down by some bigger punk and he says “500” … pause … punk asks “500 whadd ?” .. Diesel retorts “500 fights.. it takes 500 street fights to learn the craft”  .. or something to that effect.

Maybe we need to wear our failed startups as a sign of pride, a tattoo to show off .. because a failed startup is going to teach a developer or entrepreneur an incredible amount in a short space of time – its the perfect learning environment, where you are engaged, get to use cool stuff, change hats, have mutable roles, and are challenged beyond your comfort zone on a daily basis.

We could be getting close in Melbourne and Sydney to that magic number, be it 500 or otherwise, where we have enough of a pyramid of talent to fuel a viable startup ecosystem.  recycle, dammit.