The completely bogus Zappa vs Sinatra test
I have recently been playing with V8 Javascript interpreter - if only because I really dislike JS's syntax. The reason was CoffeeScript, which is more palpable to me.
After going through some loops to get everyting installed poperly i created the simplest possible website using both tools. The idea was to quickly and dirtily test the raw speed of serving a web page. After all both Sinatra and Zappa are almost "bare metal" DSLs.
After seeing V8 kick the living daylights out of Ruby 1.9.2 in The Computer Language Benchmarks Game (Which is somewhat more serious than this test - but still just as misleading), I decided to see the Zappa fly past Sinatra in awe.
After seeing this, it was more of an OUCH than AWE.
Here is the simplest zappa application I used:
#File: cuppa.coffee
get '/': "Hello World!"
And it's Sinatra counterpart:
#File: cuppa.rb
require 'sinatra'
get('/') { "Hello world!" }
It should be noted, that this is not a real, proper website. In fact, if you create a website like this, you are liable to get a wedgie and a wet willie at the same time. From a nerd, not even a bully.
Here are the httperf results:
Zappa:
# httperf --client=0/1 --server=localhost --port=5678 --uri=/ --send-buffer=4096 \
# --recv-buffer=16384 --num-conns=100 --num-calls=100
Maximum connect burst length: 1
Total: connections 100 requests 10000 replies 10000 test-duration 10.195 s
Connection rate: 9.8 conn/s (102.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 76.7 avg 102.0 max 214.4 median 80.5 stddev 42.1
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 100.000
Request rate: 980.8 req/s (1.0 ms/req)
Request size [B]: 62.0
Reply rate [replies/s]: min 976.5 avg 988.1 max 999.7 stddev 16.4 (2 samples)
Reply time [ms]: response 0.8 transfer 0.2
Reply size [B]: header 279.0 content 12.0 footer 0.0 (total 291.0)
Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 6.00 system 4.16 (user 58.9% system 40.8% total 99.7%)
Net I/O: 338.7 KB/s (2.8*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
Sinatra:
# httperf --client=0/1 --server=localhost --port=4567 --uri=/ --send-buffer=4096 \
# --recv-buffer=16384 --num-conns=100 --num-calls=100
Maximum connect burst length: 1
Total: connections 100 requests 10000 replies 10000 test-duration 8.131 s
Connection rate: 12.3 conn/s (81.3 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 68.0 avg 81.3 max 118.3 median 71.5 stddev 12.8
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 100.000
Request rate: 1229.9 req/s (0.8 ms/req)
Request size [B]: 62.0
Reply rate [replies/s]: min 1224.7 avg 1224.7 max 1224.7 stddev 0.0 (1 samples)
Reply time [ms]: response 0.8 transfer 0.0
Reply size [B]: header 147.0 content 12.0 footer 0.0 (total 159.0)
Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 4.59 system 3.50 (user 56.5% system 43.1% total 99.6%)
Net I/O: 265.4 KB/s (2.2*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
This was a test making 100 requests with 100 connections each.
How about making a normal web page with a title and a couple of lorem ipsum paragraphs. Notice we are not using any fancy templating and whatnot. This is more or less about the speed of serving, not about the speed of a templating engine.
Zappa:
#File: cuppa.coffee
get '/': '''
<!DOCTYPE html>
<html>
<head><title>Sorry, we'll be back soon!</title></head>
<body>
<h1>Sorry, we'll be back soon!</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed sed quam eu nunc rutrum luctus ut vestibulum massa. Fusce fermentum volutpat ullamcorper. Phasellus est felis, pretium ac viverra non, tincidunt id nulla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Maecenas neque sapien, tincidunt et pretium a, ullamcorper at lacus. Mauris faucibus lacus non dui viverra vitae lobortis dui fringilla. Sed eget augue nec ipsum eleifend porta sed et mi. Aenean sit amet nulla ac nisl aliquam malesuada non id justo. Ut varius erat vel justo semper congue vel eu velit. Praesent ac dui eget urna ornare eleifend nec id sem.</p>
<p>Duis at tempus lorem. Aliquam erat volutpat. Praesent auctor rhoncus egestas. Suspendisse erat lectus, lacinia sit amet varius sed, interdum at elit. Vivamus ac volutpat odio. Nullam dolor leo, sagittis a fringilla vel, sollicitudin eu sapien. Etiam pretium, urna vel semper luctus, diam metus facilisis augue, ac placerat ante nunc molestie sem. Suspendisse potenti. Vivamus ligula nisi, cursus id volutpat quis, tincidunt non neque. Curabitur neque tortor, ultricies quis elementum eu, aliquam id lectus. Vestibulum et odio at purus faucibus blandit.</p>
</body>
</html>
'''with the following result:
# httperf --client=0/1 --server=localhost --port=5678 --uri=/ --send-buffer=4096 \
# --recv-buffer=16384 --num-conns=1 --num-calls=10000
Maximum connect burst length: 0
Total: connections 1 requests 10000 replies 10000 test-duration 23.197 s
Connection rate: 0.0 conn/s (23197.4 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 23197.4 avg 23197.4 max 23197.4 median 23197.5 stddev 0.0
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 10000.000
Request rate: 431.1 req/s (2.3 ms/req)
Request size [B]: 62.0
Reply rate [replies/s]: min 397.2 avg 428.2 max 470.6 stddev 36.9 (4 samples)
Reply time [ms]: response 0.9 transfer 1.5
Reply size [B]: header 281.0 content 1382.0 footer 0.0 (total 1663.0)
Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 13.62 system 9.53 (user 58.7% system 41.1% total 99.8%)
Net I/O: 726.4 KB/s (6.0*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
Sinatra:
#File: cuppa.rb
require 'sinatra'
get '/' do
"<!DOCTYPE html>
<html>
<head><title>Sorry, we'll be back soon!</title></head>
<body>
<h1>Sorry, we'll be back soon!</h1>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed sed quam eu nunc rutrum luctus ut vestibulum massa. Fusce fermentum volutpat ullamcorper. Phasellus est felis, pretium ac viverra non, tincidunt id nulla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Maecenas neque sapien, tincidunt et pretium a, ullamcorper at lacus. Mauris faucibus lacus non dui viverra vitae lobortis dui fringilla. Sed eget augue nec ipsum eleifend porta sed et mi. Aenean sit amet nulla ac nisl aliquam malesuada non id justo. Ut varius erat vel justo semper congue vel eu velit. Praesent ac dui eget urna ornare eleifend nec id sem.</p>
<p>Duis at tempus lorem. Aliquam erat volutpat. Praesent auctor rhoncus egestas. Suspendisse erat lectus, lacinia sit amet varius sed, interdum at elit. Vivamus ac volutpat odio. Nullam dolor leo, sagittis a fringilla vel, sollicitudin eu sapien. Etiam pretium, urna vel semper luctus, diam metus facilisis augue, ac placerat ante nunc molestie sem. Suspendisse potenti. Vivamus ligula nisi, cursus id volutpat quis, tincidunt non neque. Curabitur neque tortor, ultricies quis elementum eu, aliquam id lectus. Vestibulum et odio at purus faucibus blandit.</p>
</body>
</html>"
endWith the following result:
# httperf --client=0/1 --server=localhost --port=4567 --uri=/ --send-buffer=4096 \
# --recv-buffer=16384 --num-conns=1 --num-calls=10000
Maximum connect burst length: 0
Total: connections 1 requests 10000 replies 10000 test-duration 8.073 s
Connection rate: 0.1 conn/s (8072.8 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 8072.8 avg 8072.8 max 8072.8 median 8072.5 stddev 0.0
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 10000.000
Request rate: 1238.7 req/s (0.8 ms/req)
Request size [B]: 62.0
Reply rate [replies/s]: min 1236.5 avg 1236.5 max 1236.5 stddev 0.0 (1 samples)
Reply time [ms]: response 0.8 transfer 0.0
Reply size [B]: header 149.0 content 1398.0 footer 0.0 (total 1547.0)
Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 4.51 system 3.52 (user 55.8% system 43.6% total 99.5%)
Net I/O: 1946.4 KB/s (15.9*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
This time I used slightly different settings. I opted for 10000 calls, but a single connection each. The results are rather ghastly. Sinatra is so much faster, that I had to redo this several times:
All tests were preformed on my Lenovo x300 laptop running Mint Linux, rolling Debian edition (64bit). This was not done in a controlled server enviroment over the network, but rather on a desktop machine, running X and whatnot. The results are therefore only valid for this particular machine, which boasts an Intel Core 2 Duo L7100 (800Mhz FSB, 4MB Cache) processor. This is ULV people. Not somehing found in your average server. It's preformance is subpar in terms of speed, so your Xeon will do much better.
It's also important to note, that Ruby was the vanilla installation of Ruby, while I had to compile V8 and Node.js using (again) Debian's vanilla GCC and the tools.
I had a slight itch to port Soxer to Javascript just for preformance reasons. This itch has now been scratched and I can move on. I will however implement a custom HAML filter for Coffeescript, which means hosting a V8/Node.js on my server anyway.
Please, do not choose your developement platform or tools based on this review. (But if you do - and get sinking rich - you can buy me a beer.)
blog comments powered by Disqus
News