Just a quick post, thought I'd let the world know that I'm changing the main goals I had for readinfra. I sat down yesterday and started to plan out bits of how I was going to do readinfra, what the design was going to look like among things when I thought to myself, I've not actually checked out if someone else has even done this. Turns out quite a few have (typical), one that stuck out was readbag, it's written with google app engine too, nice eh? So with simlar services out there I've decided to make readinfra about hitting some nice goals I'd like to achive.

For one I'm going to develop the site using the merb framework so theres a goal already get good at merb and maybe become one of those big sites using merb? Other gloals....well shall I just list them?

1. Simple, clean and fast design!

I've decided to keep the design very simple and clean, as too focus more on your links and content instead of fancy graphics. Also by being clean I'm hoping that the design will be fast! Not having lots of images and making sure I cut right down on bloated javascript prettyness I'm hoping it'll make the site snappy.

2. Rapid development

I want to get this done quick, I've read about quite a few of these modern type services being done in 24 hours! That's not my goal as I don't think I'd manage that (far to busy... yes...) but I want to only spend about a week of development time getting it up to a nice useable level. The time will be spread over a few weeks or so though.

3. Learn, develop and master Merb

I'm looking at merb as another ruby based alternative to rails -which although very nice might be an overkill for certain things as it comes with a lot of stuff. Merb is different to rails though it takes more of the approach of adding what you need instead of having it all thrown in there. I ran some simple benchmarks the other day by writing the same app in rails and merb.

This app simply pulled some records out the same database, they both had the same layout and pretty much the same code and here are the results I got from running apache benchmark (ab) making 10000 requests with a concurency of 100.

Rails development mode: 22.98 req/sec
Merb development mode: 144.62 req/sec
Rails production mode: 151.18 req/sec
Merb production mode: 222.90 req/sec

req/sec = requests per second.

As you can see merb seemed quite a bit faster so looking forward to some development with merb.

3 goals will do for now, I'm sure I'll have much more as I go though the development stage... so stay tuned join the beta and hopefully you'll enjoy the service.

So the other day I decided to switch from using Apache to Nginx not because apache isn't any good, far from it.  Nginx just uses much less ram and when you're on a fairly small slice, using up more ram just isn't good. I was going to benchmark speed and things between apache and nginx but I forgot to take some readings before stopping the apache service and installing nginx so we'll skip over that idea like the thought never came into my head.

I'll mention a few things I like about nginx first:

1. It's Russian -Being Russian it makes me feel like a communist, which is fun. It also reminds me of being build to take all things thrown at it.
   
2. It uses a lot less memory - As I said before, not really done much benchmarks so these were some quick figures from 'top' but I had apache running with about 5 instances each using 2-3% of ram even at 2% each that's still 10% of ram being used just for apache... with nginx I appear to have 2 instances running both using 0.2% of ram, so 0.4% in total - which is a nice difference.
3. Faster page serving - Now this is meant to be a server fast at serving static content so its not just be me but I find the site much quicker compared to when it used to be on apache (all the sites in fact). I'm not to sure if I just think it's quicker, the slice has more ram free so never going into swap or.... it is just a lot quicker at serving pages.

Installing nginx

I follow pretty much most of these from slicehost's articles but I didn't find an article on how to setup the sites-available and sites-enabled type directories for Debian (if you compile from source), so I'll cover that here.

On Debian the package manager contains a rather outdated version of nginx so I'd recommend installing it from the latest source. we'll start by making sure you have all the required library's:

sudo apt-get install build-essential sudo apt-get build-dep nginx sudo apt-get install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev

When you're ready you need to obtain the latest source from the nginx website, the latest stable when writing this was 0.6.31.

cd ~ mkdir sources && cd sources wget http://sysoev.ru/nginx/nginx-0.6.31.tar.gz tar xzvf nginx-0.6.31.tar.gz cd nginx-0.6.31

There are a few options we need to set when compiling from source. I wanted my nginx configuration files in /etc/nginx instead of the default, so you may ignore this option (--conf-path) if you prefer the defaults.

We also need to change the location of where nginx puts it's binary files using the --sbin-path option because by default this path is set to /usr/local/nginx meaning the binary files would be in /usr/local/nginx/sbin/nginx which won't be in our PATH setting, this option will allow the binary files to be placed where it can be found.

The last option will compile nginx with the ssl module, so we can have ssl conections.

sudo mkdir /etc/nginx ./configure --sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf --with-http_ssl_module

When the configuration has finish you can compile and install it using make:

make sudo make install

This should hopefully of installed nginx into /usr/local/nginx and the configuration files into /etc/nginx.

When compiling from source we need to make the /etc/init.d script ourselves otherwise we'll have no way of starting, stopping and restarting nginx. Slicehost have a script on one of their articles that should be suitable here, so we'll download it and move it to the right directory.

cd ~ wget http://articles.slicehost.com/assets/2007/10/19/nginx chmod +x nginx sudo mv nginx /etc/init.d/nginx

Then finally add it to all the default run levels.

sudo /usr/sbin/update-rc.d -f nginx defaults

You should now be able to start nginx and browse to it http://127.0.0.1 (change with your servers IP). If all is working OK you should see a message that says "Welcome to nginx".

sudo /etc/init.d/nginx start

Configuration for Nginx

The compiled version of nginx doesn't quite have the structure you would have if you'd installed it from apt-get, the package manager sets up a familiar structure to Apache having the /etc/nginx/sites-available and /etc/nginx/sites-enabled directories, I find this to be a good way of organising all your virtual hosts so lets add that feature.

sudo mkdir /etc/nginx/sites-available sudo mkdir /etc/nginx/sites-enabled sudo vim /etc/nginx/nginx.conf

After we've created the two directories we'll open the nginx.conf file so we can add the following line which will include our virtual host files.

include /etc/nginx/sites-enabled/*;

I was unsure the best place to put this so I placed mine after the server {} block, like this.

http { ...... server { ...... } include /etc/nginx/sites-enabled/*; }

Once you have done that we are ready to setup our virtual host, for this well proxy all the dynamic content to mongrel and serve all static content using nginx.

sudo vim /etc/nginx/sites-available/site_name

You'll want to replace the site_name with the name of your site. Into this file paste the following:

#Setup the location of your mongrel servers. upstream give_proxy_a_name { server 127.0.0.1:8000; server 127.0.0.1:8001; } #Rename www.site_name.com to site_domain.com #Remove this if this is something you don't want. server { listen 80; server_name www.site_name.com; rewrite ^/(.*) http://site_name.com/$1 permanent; } server { listen 80; server_name site_name.com; #Setup nginx logs. access_log /path/to/rails/project/log/access.log; error_log /path/to/rails/project/log/error.log; root /path/to/rails/project/public/; index index.html; location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect false; if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; } if (-f $request_filename.html) { rewrite (.*) $1.html break; } if (!-f $request_filename) { proxy_pass http://give_proxy_a_name; break; } } }

Replace site_name.com with your domain. You can also rename give_proxy_a_name to something else, the name of your site (e.g. 27smiles) should be fine here.

So what does some of that mean? Well the log and root stuff are pretty straight forward, they define where to put the logs and the what the root folder for the domain is, which in the case of rails will be the public folder within the rails project.

upstream give_proxy_a_name { server 127.0.0.1:8000; server 127.0.0.1:8001; }

This is where you setup the mongrel cluster, for each mongrel server you have running you will need to put a server directive here nginx will then load balance between the servers.

location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect false; if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; } if (-f $request_filename.html) { rewrite (.*) $1.html break; } if (!-f $request_filename) { proxy_pass http://give_proxy_a_name; break; } }

There are 3 IF statements to this bit, the first checks to see if an index.html file exists and if it does then nginx will serve it rather than passing it onto mongrel.

The 2nd IF does pretty much the same but it looks for a html file with the same name as the request. So say you requested site_name.com/about it will attempt to find a file called about.html in the public folder of the rails project and serve it rather than passing it onto mongrel. This means that if you're using page caching in rails you can skip mongrel for cached pages - very good on performance.

The 3rd IF makes sure the file doesn't exist before passing the request onto mongrel. For example say you requested site_name.com/images/logo.png nginx would attempt to locate the file and serve it rather than passing the request on to mongrel, this means you can serve all your static content using nginx instead of mongrel which is much much quicker.

When you've setup your virtual host you need to create a symlink to the sites-enabled folder in order for it to work.

sudo ln -s /etc/nginx/sites-available/site_name /etc/nginx/sites-enabled/site_name

And then finally restart the nginx server.

sudo /etc/init.d/nginx restart

If all went OK you should be able to browse to your domain name and see your rails project. If not then make sure you created the link to the sites-enabled folder and also that you actually have some mongrel instances running on the IP's you defined (I think you'll get bad gateway messages if it can't find mongrel).

If you need help setting up mongrel head over to slicehost's article page they have some good articles on how to do it.

Extra configuration options

There are a few extra configuration options in nginx that I use on 27smiles; gzip, expiry headers, on another site I also use HTTP authentication. I thought I'd list them here for some people.

If you're looking to setup gzip compression on nginx then add the following.

gzip on; gzip_min_length 1100; gzip_buffers 4 8k; gzip_proxied any; gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

If you're looking to set some expiry headers to images, etc in nginx try this.

location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|mov)$ { expires 30d; break; }

And finally if you want to password protect domain try this.

auth_basic "Restricted"; auth_basic_user_file /path/to/htpasswd/file;

If you need any help or have any questions feel free to contact me.

Off to see Kerry at aber this weekend so thought I'd pump out a quick youtube Friday before I head off.

First off I thought I'd start with the Floyd, this is from division bell, which is one of my favourite Pink Floyd album.

Next up a bit of Thom, not to sure if I've posted this before - I have yet to import posts from the old blog. I'm sure you can survive having to watch it again.....after all it is Thom :)

Finally I thought I'd post this video I found, I was orginally going to find Bob's version, and then the one thats playing at the end of the last season of Battlestar Galactica but instead I found this insane child playing it like a pro. Enjoy (All along the watch tower btw).

Lately I've become fanatical over web frameworks, mainly Django, Rails and CakePHP. Something about them just clicks, maybe its because I can go back to a rails app months after I've written it, even with no comments, and just know where each file is likely to be and where to find a certain method. Also I can create CRUD (Create, Read, Update and Delete) functionality in seconds as most frameworks provide some form of scaffold, something that would take time to produce without a framework in something like PHP.

If there is one thing that pisses me off about frameworks - it's the people who don't like them. If like me you've read a lot of people talking about how crap frameworks are? How they don't see the point: "What's the point in a framework? I can just find all the different library's I need such as templating, databases and include them in my app". Why? Why would you want to waste your time including a database class, setting it up and then going to find a template engine to include and then setting that up too! When the framework will just give you that from the word go. It just confuses me. Then there is the fact that with most frameworks you get nice clean URL's, free without having to do anything. try doing that in your simple PHP app without doing some form of mod_rewrite YOURSELF.

Most frameworks will log errors, show debug information (in development mode), give you the option of simply editing a file (not having to set it up too) to get nice 404/500 type errors. Something that would require you to make your own form of exceptions, your own error logging if you were not using a framework, which would do it all for you.

Another thing that pisses me off about not having a framework - is validating form inputs. Argh, it annoys me so much having to do lots of if statements to check if a field is empty, having to do a query to check a field isn't already taken. In a framework, for example rails I just have to do:

validates_presence_of :title

Tell me again why people prefer to spend ages writing if statements and queries to check form data?

People will always try to say "I don't like being told how I should code" - Fuck me, yes maybe...maybe there are some standards you have to stick to in frameworks and if you're totally mentally retarded you might think this is horrible idea to have to agree on a naming convention for the benefit of others. It's like these people have never worked with other people? At work we've had entire meetings just agreeing on a standard form of code to use, have you ever tried submitting a PEAR class?! talk about anal. I've found apart with some naming conventions you can pretty much code however the hell you like in a framework - much like you can if you were not using a framework!

From my experience so far I've never had a website that has not fitted the framework type model... now I'm sure there are loads of sites that would not work at all in a framework and the anti-framework people will be the first to tell you but that's fair enough if it doesn't fit the model then you don't have much choice. But these people do it out of choice?! Are they better than us mortals? Do they know something we don't? No! from my experience the people who shoot down using a framework to speed up development, to help bring out creativity, to help with debuging, these people are generally scared to learn something new, scared to well...learn. They refuse to even look becuase they know it'll be something else that's not what they know, even if the current method they use is rubbish.

All this technology, all the advances in medicine we have, space travel, you name it. It all comes from people wanting to discover, people wanting to learn. So I beg you, don't just push the framework out the window as though it's all 'hype' and doesn't have any real world use.. please please please just give it a try. There's a reason it's popular, even among the big names, because frameworks are damn good.

Man have I been busy the last two weeks what with travelling and doing projects for other people, feels like I never get time to write a blog post - which of course will be my excuse. You see I get home from work have some dinner, do a bit of a project and before you know it the night has gone, gone like the acting skills of Keanu Reeves in, well anything?

I've missed youtube Friday twice now and quite frankly that just needs to be sorted out right now. So bring on youtube Friday. Before I post some videos I'd like to bore you, come on entertain me for a second or too - I'm of course talking about what I'm up to and stuff.

Lately I have been busy doing a website for someone which I initially started in cakePHP and switched to rails - don't ask me why!? Maybe it's because I'm a fan boy (GO RAILS, GO APPLE!) and like a good fanboy I do as I'm told. Not to say I don't like cake, I love cake! Frameworks in general are just a really neat idea.

I did manage to finish the project from scratch in a weekend though - which was handy and I even tried JRails to replace prototype with jquery, much much better and all the helpers and RJS I normally use still work.

While we're in that framework sort of area - I'm going to relearn the limited python I know and do readinfra.com in django and here's the twist I might just record screencasts of what I learn and how I learned it right from the beginning so if anyone else wants to pickup python and/or django feel free to watch along.

For the project I did at the weekend I decided to switch to using GIT to revision control it, and I has to say I do really like it. Yes! More fanboyisum but I just love how quick it is, how easy it is to do branches. I actually use branches now with SVN I'd always create a branches folder but did I ever use it? Nope. For readinfra I'm going to try Mercurial (HG) though see if I perfer that to git, has better windows support and while I personally don't care about that as I don't ever see myself trying to run rails/python apps on windows and pretty much always developing on my Mac, others however may differ so easier for team projects.

Enter youtube fun

The first video I'm posting is of good ol' Thom. Thought I'd start off with a quite video but doesn't Thom just look the perfect picture of strange folk at the start, when he takes a nice glare at the crowd. Anyhow enjoy - oh and shutup Jools.

The second video is from someone accross the pond, so to speak from Thom. Stephen Malkmus, singing in a lift, while clapping.

I'm a new soul, I came to this strange world, Hoping I could learn a bit bout how to give and take, But since I came here, felt the joy and the fear, Finding myself making every possible mistake.....macdell air.

Ah, Tony and Bush - at the gay bar?

So....I've got a project coming soon hopefully... when the free time is given to me. I won't release information about it leave it as a nice surprise. I'm hoping it will be of some use to people but who knows eh? :). A few lucky people could have some beta access, if you want to try the beta then email me.

I went to intergrate Syntaxhighlighter into the site tonight, something I'd forgotten to do in the redesign and I thought it would be nice if I could just click a button in tinymce and paste in the code, select a few options and click insert. Which of course is entirely possible so I've written a tinymce plugin to do just that.

I did encounter one little bug writing the plugin. When inserting the pasted code it keept repeating itself if there was currently no content in the body of the textarea (tinymce). I finally managed to fix it by putting a space at the end of the variable I was trying to insert (and with the power of highlighting here's what that looked like).

textarea_output += '</textarea> '; /* Note space */ tinyMCEPopup.editor.execCommand('mceInsertContent', false, textarea_output);

You can download the plugin here:
Download (ZIP) | Download (GZIP) | Download (RAR)

Getting it to work

First you need to download Syntaxhighlighter and get that working by including the CSS file and Javascript files. I combind all of the syntaxhighlighter javascript files into one on 27smiles, you can get it here.

Extract the plugin

Next you need to extract the plugin to your tinymce plugin directory. When extracted it should be something like this:
/path/to/tinymce/plugins/syntaxhl

Configure tinymce

Finally we need to configure tinymce to use our plugin we also need to stop tinymce from stripping out <textarea></textarea> html tags as this is needed for Syntaxhighlighter.

tinyMCE.init({ theme : "advanced", plugins : "syntaxhl", theme_advanced_buttons1 : "bold,italic,underline,undo,redo,link,unlink,image,forecolor,styleselect,removeformat,cleanup,code, syntaxhl", theme_advanced_buttons2 : "", theme_advanced_buttons3 : "", remove_linebreaks : false, extended_valid_elements : "textarea[cols|rows|disabled|name|readonly|class]" });

We tell tinymce to use our plugin by adding syntaxhl to the plugins list and also adding the sytnaxhl button to the buttons list also note that extended_valid_elements contains the textarea tag this tells tinymce to not remove it.

If all went well you should see a new highlighter icon button in tinymce and when you click it you should get a dialog popup allowing you to insert code into your content.

If you encounter any bugs or have any problems getting it to work please contact me I'd be happy to help/fix bugs. Please note that I've only briefly tested this on Safari/IE7/Firefox 2/Firefox 3b5 and a quick blast on IE6 and from first tests it seemed to work. Also I was unsure if you have to stick to any standards when writing plugin's for tinymce so it might not be the correctly way of doing it.

Wow, is what you're thinking eh? A final return of the youtube Friday, my weekly collection of music video's that I've listened to/watched throughout the week? Well yes... but I've decided to open the weekly event to more than just music...I've decided folks to open it up to any youtube video (what you're crazy!), that's right any video. Now before you all go shouting and all the craziness makes me want to become Amish let me tell you this tonight's videos are of a music nature.

Sleeping is the only love

How fucking creepy are his eyes in this video? The first video tonight is from one of my favrioute musical bands. Silver Jews. This song is off their 2005 album Tanglewood numbers (track number 6).

Radiohead - 15 Steps (Seven)

This video is funny with Thom in a box, yay. 15 Steps is from their latest album In Rainbows and this video was on one of their webcasts. You can checkout greenplastic.com or ateaseweb.com for some Radiohead fansites.

What What, in my butt Lets fighting love

This video was on the latest south park, which was funny as normal the south park boys basically trying to make it big on the internet and this is the kind of video that's big on youtube. Ok so some stupid ass company removed all the videos from youtube that I could find, so for now another song from south park, lets fighting love. Enjoy and happy youtubing.

New website

I've finally fixed the website, from that file I deleted. True it wouldn't of required that much effort to fix but I thought I'd give the site a re-code/design while I was at it - I seem to have a habit of that but anyhow since I've improved on my Rails skills since the last code this build went much smoother which leads nicely into the next change. The layout.

I'm sure you have noticed the new layout but I bet you did know that it in fact changes between a night theme and a day theme at 6AM or 6PM.... but if you prefer one theme over the other why not click one of those little icons in the top right? They change the theme to whatever one you choose for that session (while the browser is open) you can even switch back to the time based theme if you prefer that.

You'll also notice the flickr, twitter and last.fm integration on the right of the page. Apart from twitter they are all pulled in from an RSS feed and then using fragment caching with memcache in rails to do some time based caching so the flickr feed is updated every hour and the last.fm one every 5 minutes. I will talk more about how I did this in a later post.

The twitter feed is updated/imported using their javascript code so that should be real time. Oh and about twitter I've finally decided I'd get one after months of thinking hmmm is there much point in telling people what I'm doing all the time? Well I thought meh why not? People don't have to read it and its something all the 'cool' kids are doing.

Holiday

So for the past two week I've had some time off work and was lucky enough to spend 5 days in Edinburgh and 3 days in London with my girl friend (Kerry). Both trips went smoothly and a I thoroughly enjoyed both (I even enjoyed London :o).

Edinburgh is both amazing and beautiful me and Kerry had a fantastic time here looking around both the city, art galleries and museums of which there are lots! We took a Flybe flight up to Edinburgh from Birmingham on a small plane which was nice. When we arrived in Edinburgh they had a very handy and excellent bus service from the airport which took us all the way into the centre, the buses run every 10 minutes - I really did think this was excellent. We stopped in the Royal Garden Apartments which was a very nice apartment. The apartment was but 5 minute walk from Prince's Street and any shops we needed also being close to most of the attractions. Oh and the national portrait gallery was right opposite the apartment so of course we went there first =) (left image was view from the window).

On the second day we went to Edinburgh castle, although I'd been before I was quite young so I only remembered bits so going again was a very fun day out, it also meant I got to take lots of pictures and blend in with all the cute Japanese tourists taking all their pictures. The castle was quite busy as you expect though there were a lot of school trips there with all these Scottish kids, I love the Scottish accent its cute.

On the third day we decided to visit all the galleries - national, modern, etc... these were spread all over the city so it took all day to get around. They were quite interesting but I'm sure Kerry enjoyed/appreciated them more so than I.

On the fourth day we went to the Museum of Scotland and the Royal Museum, which are attached, these were both very interesting and fun :), while being quite big and hard to navigate around at the same time.

The 3 days we spent in London were very fun too, again we visited the art gallery and the natural history museum  which were both very big and full of interesting paintings and animals the real excitement of course for me was the science museum :D getting to play with all the interactive stuff like a little child, amazing amounts fun indeed. Although we did get evacuated from the building for some reason, and I'd love to know why lucky for us though we were in the last room.

The tube was not to bad at all, we found our way around with ease and although it got a little busy now and then majority of the time it was pretty empty. Also for some strange reason unbeknown to me I like the smell at the tube station, must be all that unclean air plus the tube kinda reminds me of the 60's, maybe its because the icon looks all retro.

So all in all the holiday's went very well and I enjoyed both lots and lots, although going back to work was a shock to the system.