i made a simple web site with rails, it just serves a web page and sends a query to mysql and shows it - just a simple web application-. i ran all benchmarks from a local networked ubuntu, so benchmarking should not be effecting performance. i ran each benchmark 3 times - to warm it up - and then ran again 3 times, these numbers are last three runs.
i used ruby, mysql from ips stack. i did same tests with thin and mongrel.
first i ran test with 1 process, and then i used nginx to reverse proxy to 4 processes. then i rebooted the machine, and started over in ubuntu 8.10. ruby/mysql/nginx was installed by synaptics - ruby version differs btw, ubuntu comes with ruby 1.8.7 installed and opens solaris (2008.11) comes with ruby 1.8.6.
so as long as the machine is identical - its the same laptop - these numbers may give you a clue.
i am using zones in solaris, first i ran only one thin process, and benchmarked with ab.
ab -n 1000 -c 100 http://192.168.1.209:9690/chat/
single process - thin
Requests per second: 172.14 [#/sec] (mean)
Requests per second: 187.11 [#/sec] (mean)
Requests per second: 178.25 [#/sec] (mean)
and then i thought if i do this in global zone what will change...
Requests per second: 188.44 [#/sec] (mean)
Requests per second: 188.57 [#/sec] (mean)
Requests per second: 173.15 [#/sec] (mean)
there was not much difference, so zones are really lightweight. so i didnt repeat the tests in global zone.
anyway, 4 thin process, upstreamed from nginx
Requests per second: 229.80 [#/sec] (mean)
Requests per second: 249.45 [#/sec] (mean)
Requests per second: 267.92 [#/sec] (mean)
now i wanted to see how mongrel is compared to thin on solaris.
mongrel 1 process
Requests per second: 126.04 [#/sec] (mean)
Requests per second: 121.87 [#/sec] (mean)
Requests per second: 119.53 [#/sec] (mean)
and mongrel 4 process on opensolaris
Requests per second: 182.07 [#/sec] (mean)
Requests per second: 177.18 [#/sec] (mean)
Requests per second: 176.31 [#/sec] (mean)
then i rebooted the machine into ubuntu and started again
thin 1 process
Requests per second: 253.94 [#/sec] (mean)
Requests per second: 242.28 [#/sec] (mean)
Requests per second: 246.67 [#/sec] (mean)
thin 4 process upstreamed by nginx on linux
Requests per second: 336.30 [#/sec] (mean)
Requests per second: 326.03 [#/sec] (mean)
Requests per second: 320.67 [#/sec] (mean)
mongrel 1 process
Requests per second: 196.33 [#/sec] (mean)
Requests per second: 196.13 [#/sec] (mean)
Requests per second: 198.44 [#/sec] (mean)
mongrel 4 process upstreamed by nginx
Requests per second: 238.31 [#/sec] (mean)
Requests per second: 244.72 [#/sec] (mean)
Requests per second: 233.49 [#/sec] (mean)
so here are the graphs
thin 1 process linux vs. open solaris
thin 4 process linux vs. open solaris
mongrel 1 process
mongrel 4 process
here is the configuration in two operating system
ubuntu linux (8.10)
uname -a :
Linux aybars-laptop 2.6.27-11-generic #1 SMP Thu Jan 29 19:28:32 UTC 2009 x86_64 GNU/Linux
ruby :
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
mysql :
mysql Ver 14.12 Distrib 5.0.67, for debian-linux-gnu (x86_64) using readline 5.2
gems :
thin (1.0.0)
eventmachine (0.12.6)
ventmachine (0.12.6, 0.12.2)
mongrel (1.1.5)
Rails 2.1.1
open solaris (2008.11)
uname -a :
SunOS opensolaris 5.11 snv_101b i86pc i386 i86pc Solaris
mysql:
mysql Ver 14.12 Distrib 5.0.67, for pc-solaris2.11 (i386) using readline 5.1
ruby:
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-solaris2.11]
gems:
thin (1.0.0)
mongrel (1.1.5)
eventmachine (0.12.6)
mongrel (1.1.5)
rails (2.1.1)
in conclusion, i have several opinions about this results
1. zones are really lightweight and great
2. two operating systems are installed from stack - no optimization, no hand compiling etc.. so if you install just inserting a cd and boot then ubuntu beats hard. maybe its just because of ruby version or something with flags used to compile ruby, but i guess an operating system is not just the kernel, the components must be optimized and as new as possible.
3. i really am curious why the hell my kernel reports as i386 on solaris, shouldn't it be x86_64 or similar. it didn't ask me about anything, and there is only one iso.
anyway stack version of both systems, ubuntu default installation beats hard open solaris, so i think i will stick to my ubuntus on our servers.
and btw, please take this benchmarks just, as out of the box experience, none of them are optimized.
just googled and saw open solaris says i686 in uname, ~$ isainfo
ReplyDeleteamd64 i386
http://opensolaris.org/jive/thread.jspa?messageID=330405
so benchmarks are ok both kernels are 64bit
The way you made that test, makes it a big waste of time.
ReplyDeleteYou're comparing stupid things with stupid things.
1st ab is not usefull when dealing with web loading, a constant load on the same page with no latency is not something realistic and does not really involves os specifics actions like network optimisation, scaling, multiprocessing ...
2nd your mysql run on a filesystem this FS is ZFS on OpenSolaris and I assume ext3 on Linux, ZFS is more robust but at a price it consumes memory and is far slower than UFS, which is closest to ext3 in term of features
3nd You've made your test on a laptop, Linux have 10years behind supporting exotic hardware, Solaris/OpenSolaris don't, but I'm sure your servers are not laptops
So If your goal is let's say 300 req/s, your conclusion is completly wrong.
But this was not your goal, your goal was to test the same page same sql request, so yes we could say: the default rails configuration of OpenSolaris is slower than default Ubuntu configuration, when asking thousand times the same page with ab... very usefull.
That's how stupid test makes stupid choice, and a good demonstration why developers are not architechts
The only thing that makes a bit of sens is the Zones comparaison.
As a 10years sysadmin I've observed Solaris outperforming Linux under high load, performing threading, SMP much better.
What is the point of benching different versions of a software on different OS ?
ReplyDeleteLet's compare a ferrari on a highway with a beetle in the mountain :)
It just remind me someone talking of blogs and how internet share the knowledge to everybody... do you really think every blog has knowledge ?
@akhen, well i commented a couple of times, but again to make it clear, this is just a quick test, with default installation from a new comers point of view.
ReplyDeleteif i am not clear, i'll express it again, this is just an out of the box experience, it doesnt show in any way ubuntu is better or not on production, or outperforms in heavy load.
btw, i really really loved open solaris. it just rocks. its stable, zones are something that i have wanted to use for ages, as a developer i cant even express how happy i am. i can boot a virtual machine in 2 seconds, and thanks to ZFS i can take a snapshot of it under a second.
i dont even want to mention about dtrace, its just a revolution.
and if this will satisfy anyone, i am using netbeans with nearly 10 open projects, and its starting under 30 seconds, - which was more then a minute under ubuntu last time. so i dont think zfs is slower then ext3, at least on a daily usage.
and btw SMF is great, we are using monit on linux, to achieve same goals - restart a service when it crashes (mongrels crash), with SMF its just out of the box, and there is no delay between a crash and a restart.
now let me answer your points,
1st, imo, there is no way to simulate a production environment. ab just gives a clue. if your application runs 180 reqs/sec at a revision and 100 reqs/sec at an other revision you get a headsup. and you digg into your changes. but if you get a 180 reqs/sec in one revision and 170 at another one, its nothing, many things effects. - you may have other applications, someone is uploading a picture what ever.
so this is a tool, and if you measure the same thing, in same conditions with the same tool, you get a clue - just a clue.
2nd, you have a point, in anyway you - i mean anyone - shouldnt put the cd and just install opensolaris/mysql on production. this is same for ubuntu too. its not the right way.
but thats why i just send only one query to mysql with a table of 3 rows - so in case of speaking about io or filesystem it shouldnt be a problem becase table is already cached.
3rd. most of our servers are not too different, just commodity hardwares - most of them are amd64 dual core with a good power supply - , yes they are cheap pcs in fact - not speaking our mysql servers and load balancers, they are mission critical. when they blow we simply throw them, because they are cheap.
and yes solaris has more experience on 100k machines, but nowadays everyones moving to cheap pcs as servers. because its idk, cheap maybe...
but i think zfs has a better chance - and may outperform well - against ext3 or something similar on cheap hardware, because cheap disks crash - makes io errors - a lot, and it seems zfs doesnt care about it.
so i dont think this test is stupid, it just gives a clue, you cant put the cd and install opensolaris on your cheap servers like you do with ubuntu.
so anyways as a 10 years web application developer and a 15 years programmer, open solaris is the best - if not the bestest - operating system i have ever used on daily basis. and in conclusion yes you shouldnt use it like you use ubuntu.
@prune, i am benching two operating systems (or in fact two distros), out of the box with the same hardware. and to be frank, none of them is a beetle.
ReplyDelete