Syllogism

Wayne has a very important entry on making Neotel's CDMA phone work in Linux here on his blog, Grobister.

I'm surprised and happy that this trick worked. It's good news that the device at least uses standard serial USB. It's just unfortunate that you have to then do PPP over this as I've had plenty of ugly issues with PPP interfaces for high availability connections especially with wireless if it is less than perfectly stable. It would be nicer if they emulated a standard NDIS which you can PPPoE over, or if they eventually make an Ethernet device which doesn't require so many layers of packet encapsulation.

So I discovered after an almost unfortunate incident that my Mazda key fob is pretty much indestructible. After putting it through a full wash cycle with my jeans I tested it and it appeared to work. Still worried, I decided to open it up and make sure there was no water inside - and there wasn't. It's actually quite an impressive design, the key membrane covers the entire circuit board. The exposed space at the back where the battery touches on two contacts is sealed against the back by that same membrane which forms a gasket between potentially leaky bits and the important bits.

After once getting stuck in a car park on a rainy day with my Toyota, I'm certainly glad that more thought goes into the simple things too these days.

Yeah, I think so

Last time I checked, Durban was quite large...

I've recently had the exciting chore job of somehow putting an installation kit together. After numerous failed attempts with the Debian "bootcd" process and the Gentoo live cd, I shelved the whole idea until I could think of something better.

It finally struck me that I didn't have to scrape my eyeballs out with the prospect of GTK programming (ok, it's not that bad but it gets quite close!) and decided I'd just stick a Nevow app on the live CD bound to localhost and boot up some kind of XUL wrapper or Firefox in kiosk mode.

This was great because now I didn't have to make my own CD from scratch (the CD really just has to house something which can format the machines drives and dump some stuff on there) and I could just do some remastering of an existing live CD. I thought about using Knoppix and then Jeremy recommended I look at the Ubuntu live cd.

This too sounded great, as the Ubuntu stuff is usually pretty well documented, and in this case it most definitely was quite well documented. I don't really have to rehash the process, since the Live CD Customisation howto on the Ubuntu wiki worked great.

What I will talk about is usplash and other such fun. Basically with usplash and gfxboot, it's amazingly trivial to theme the live CD and in fact any Ubuntu install. Usplash themes are just little library files with graphics linked into them. Once again, it's all on the Ubuntu wiki at USplash Customization Howto with the exception that 'usplash-dev' which contains the required pngtousplash tool is now called 'libusplash-dev' in Hardy.

The boot splash is just a png in the isolinux directory of the CD - you just drop a new one in there. The usplash gets a bit tricky with remastering the pallets and stuff, but with a bit of trial and error I finally came right by avoiding modifications to the pallet indexes in the theme C code (there appears to be black magic about overriding the 'special' indexes). Once you've built a usplash you can just overwrite the one in /usr/lib/usplash on the Live CD and in the chroot run `update-initramfs` to rebuild your initrd.

So now I have a funky base to work an installer from. From there on I found the easiest thing to do was just fiddle with the Live CD user (/home/ubuntu) until it was right and then copy that back to the original.

The web framework debate has become as religious these days as the great language and operating system debates. Why is it though that people become so fanatical about a particular framework or language? At the end of the day our choices of language and framework should be based on merit and not on emotion.

It's not unknown that my personal choice for most web application development is Python and Nevow. I break away from the Nevow community however when it comes to database abstraction as I personally think there is more value (at least in the applications I design) to be derived from SQLAlchemy.

There is still this near war waged with PHP and Django and other frameworks though. Newcomers seem to get confused in their choice because of this. While the language itself is an important decision, there is something which people seem to overlook entirely. That is the architecture which you will be forced to adhere to by the language deployment.

If we ignore the obvious faults in PHP as a language (broken OO, poor syntax, namespaces etc etc) there is a much larger and more fundamental flaw at play. PHP is not alone in it either, think about the similarities between Perl and PHP for example. In the language there are few, but then we think about how we deploy a PHP or a Perl application we see many problems.

These problems come in the form of Apache, and people slip towards them with Python and Ruby as well. Most people consider Apache to be the webserver of choice and therefore when one is building an application you should work around Apache. These people are wrong. Other people look to alternatives like FastCGI and lighthttpd. Those people are also wrong.

Just because a web server exists does not mean you should hack yourself onto it to develop a web application. Why not write your own web server?

With PHP for example, PHP sits as a module inside of Apache. For specific resource types, Apache will pass the resource through the PHP interpreter and spit out the result from PHP. This is why you're basically printing HTML into standard output all the time. When you see a language encourage you to do this, your instinct should be to run very far away.

For one thing, as Glyph Lefkowitz correctly points out here is that a good web framework must provide a layer, beyond a simplistic template engine, that absolutely ensures you are outputting correct XML. Incorrect XHTML in a web application is an error case, not just something nerds whine about when they try to use your website. String concatenation and simply spewing garbage back to the webserver can not achieve that.

Of course, you will realise that doesn't entirely reason why web servers like Apache and lighthttpd should be ignored for web applications.

Think about what happens with a PHP web application. You upload your PHP scripts to your webserver and you start hoping that they work. They aren't running, they aren't doing anything - they are just sitting there. It is only when a client requests your PHP resource that the script will be executed. Adding to this the fact that execution flow in a traditional web application is controlled by redirecting to other resources, the client entirely controls the flow of execution. That is a bad thing. Aside from inherently opening the door to poor scalability and bad people who will break your application, it means that you have no state and poor event control.

Storing any kind of state in a PHP application is left to horrible hacks of the browser session as well as abusing your persistent storage layer.

Once you can break this mould, the endless possibilities of web applications become far more apparent. Twisted and Nevow gives you a clean web server tailored for your application, and you can keep your own state information and event control without needing the client to hit on special pages or abusing AJAX to poll resources. Think of for example instead of polling a resource with JavaScript, if your database abstraction object just had triggers such that when new entries reached it the server (not the browser) sends a request to the JavaScript on the client. While to some this might sound "real cool", it already exists in the form of COMET which can be seen in GMail for one and realised for some time now by others with Nevow and Athena. The only thing which prevents anyone else from doing the same is their poor deployment architecture.

If Java has taught us anything through Tomcat, JSPs and other horrendous beasts, it should be that when you're building new things reusability can be your biggest friend, and at the same time your biggest enemy. If the choice comes between simply implementing your own nice HTTP daemon with Twisted or one of the other libraries and hacking Apache and other daemons to do your bidding in the worst possible way, I'll chose Twisted every time. 

Well this brings an end to the recent series, and I hope people get some enjoyment from it - as much as I have writing it. IT is something I'm passionate about, it is my living and my hobby all at the same time. I enjoy working on things which are new and innovative. If I was Just Another Programmer, I'd have done my degree like a normal person and then spent my life like a drone working as a faceless employee in a big bank. I might earn a reasonable salary there, but it would never grow. The only way to move up from a standard programmers salary is to no longer be a programmer any more.

This is why I get frustrated when I see good talent being wasted, I think of the good they could do for Open Source projects. I think of the amazing solutions South Africa could be producing. We have not even begun to scratch the surface of the power and abilities of computing, and yet 95% of the people in my field are more than happy to learn all they need to learn to land a cosy job.

I don't blame them so much as I blame the incumbent IT houses. They are just downright lazy. The biggest problem is pure ignorance about the world out there. I spoke previously about hosting providers but it spreads much further than that.

Take a big corporate for example who is looking for someone to look after their IT needs. They phone up DiData or BCX who slap down some Telkom Diginet connections with a Microsoft Exchange server on each end. A few thousand rand goes into their pockets, a few hundred to their employees and 50% or more goes to overseas software vendors. They will say of course that there isn't enough Open Source support in SA (which may or may not be true), they will say that no one locally is making software - which is probably very accurate. Why aren't they doing it though?

They have the talent within their organisations to build bespoke solutions for the South African market. They have the talent to build software for the international market, but they don't.

How can you possibly blame the small IAP's for hosting their systems outside of the country while you pour money into overseas software vendors like Oracle and SAP?

Which brings me back to the hosting providers and ISP's. If I look at all the overseas operations there is a thick running theme - give the users the ability to manage themselves. Let them buy more bandwidth easily with a web interface. Treat servers and internet connectivity like a product you can stick in a shopping cart, and then configure when you get home. Linode won the virtual server rush by doing just that. People offered blank virtual machines, but Linode went one step further by giving people a vast management interface with numerous cool features - they did this because they were listening to their users. Only ONE local company has realised this, and they are my ISP of choice who I will recommend to anyone in a heart beat. I need not even plug them here, as they won MyBroadband's ISP of choice award (or whatever the award was actually called).

In contrast the rest of the South African ISP's, hosting providers and even IT companies seem intent on doing one thing alone - pissing off their users. It's not even about bad service any more, if anything there is too much service.

I don't want customer service, I don't want to talk to you when I need things done and I don't want to deal with your help desk. I just want to do it my self, plain and simple.

For all those people who talk about Hetzner South Africas "great service", I'm going to post this little snippet of chatter from this morning. (I'm Karnaugh, mithrandi is Tristan on the right hand side of this blog)

< mithrandi> Karnaugh: are you doing that from an international box?
< Karnaugh> mithrandi: yeah
< mithrandi> got a local one?
< Karnaugh> sure
< mithrandi> 'cause:
 * Karnaugh goes to screen 6
< Karnaugh> 10:  ge0-3.br1.jnb7.alter.net (196.30.228.241)             38.995ms
< Karnaugh> 11:  ge0-1.cr1.jnb7.alter.net (196.30.228.122)             38.025ms
< Karnaugh> 12:  fe2-1-1.cr7.jnb6.alter.net (196.30.229.18)           asymm 14  41.887ms
< Karnaugh> 13:  no reply
< Karnaugh> 14:  no reply
< Karnaugh> 15:  no reply
< Karnaugh> how do you do
< mithrandi> :/
< mithrandi> ok yeah
< Karnaugh> http works though
< Karnaugh> wait no it doesn't
< mithrandi> "works"
< mithrandi> as soon as you get a packet that's too large, it'll just hang :/
< Karnaugh> yup
< mithrandi> tristan@crimson:~> traceroute -F XXXXXXX 1497   
< mithrandi> traceroute to XXXXXXX, 30 hops max, 1497 byte packets
< mithrandi>  2  YYYYYYY (YYYYYYY)  0.785 ms  0.675 ms  0.697 ms
< mithrandi>  3  vlan908.hr3.jnb6.alter.net (196.31.215.66)  2.371 ms  1.704 ms  1.468 ms
< mithrandi>  4  ge4-0-0.gw2.jnb6.alter.net (196.30.156.131)  30.862 ms  2.594 ms 6.519 ms
< mithrandi>  5  * * *
< Karnaugh> ie the Athena cookie :P
< Karnaugh> mithrandi: wtf are they breaking PMTU
< mithrandi> they seem to be filtering ICMP "Don't Fragment" packets out
< mithrandi> which is really bizarre
< mithrandi> because they're not filtering, for example, ICMP Time To Live Exceeded
< mithrandi> which traceroute etc. relies on
< Karnaugh> [swearing]
< Karnaugh> why [swearing] are they filtering *anything*
< Karnaugh> [more swearing]
< Shrimp> i blame the newbie admin playing with router configs
< mithrandi> anyhow, I can't figure out where it's being filtered

Of course, the fanboys will say "Oh but I'm sure if you told them, they would fix it right away!". Yes, I'm jolly sure they will but in reality the problem is symptomatic of a worse trait - ie messing with things they shouldn't. A problem like that shouldn't ever happen in a datacenter, because they should never be filtering any kind traffic.

There have been a number of interesting discussions recently about the state of the South African IT industry.

Primarily the issue is the numerous South African sites who claim localisation as their key benefit, and whether we should care about where we host our websites.

In short, the answer is no.

On the one hand, duplicating a website for the sake of locality is pointless. When I say locality I am talking about hosting the website in South Africa or trying to turn the South African community into a market on its own, none of these sites try to cater to our many different languages. The internet is a global interconnection, and services on the internet represent globalisation. We are moving away from the idea of countries with borders when we talk about the internet, what we want is a seamless cloud of resources without barriers.

The key problem is that we are wasting our time trying to create a "local" Facebook or YouTube. We already have a local Facebook and a local YouTube, they are Facebook and YouTube and they are accessible anywhere in the world (well, except China) thanks to the wonders of TCP/IP. The only reason we care where they the traffic is coming or going to is this obscene delusion which people have been brainwashed with by Telkom. Telkom has managed to convince people that there is added cost associated with international connections.

If anything, international fibre is a cheaper commodity than local fibre. The reason for this is simply a matter of scale. It costs only a fraction more to lay two fibre optic cables at the same time as laying one, for this reason when international cables are laid they go all out laying the highest optimal capacity giving the cable the longest return on investment. This would remain a large expense if these cables were point to point only, but they aren't. SAT-3 for example terminates in numerous locations, all of whom pay for a slice of the cable. Local cables however must be solely used and resold by the group who lays them, in terms of inter-country relationships a local piece of cable is only to the benefit of that country. International connections are to the benefit of the country on both ends (they get to sell peer back to the people on the other end) as well as all the countries who are stopped at along the way. Local fibre on the other hand has to cover similarly large distances end-to-end, but each leg covers a far smaller group of users.

Of course, this is irrelevant to the end user who is still forced to pay more for international traffic for no reason at all. They should raise this with their ISP, not with content providers. Content providers want the biggest market for the least cost. The biggest market is obviously the entire planet, due to our lousy internet penetration South Africa is an irrelevantly small market. So the argument that "South African" sites should be hosted in South Africa is moot, there shouldn't be any such thing as a "South African" website, there are only global websites and that's why the W in WWW stands for World.

Why aren't we hosting in South Africa though? It should surely make the most sense to do so since purchasing hosting overseas involves high risk and dependence on the currency. Well, the truth here is that no matter how unstable our exchange rate may be, it's a far smaller risk when compared to doing business with the local hosting providers.

We can consider the local hosting market to consist of Hetzner, IS and a few smaller players (POSIX, MTN, Altonet and others). POSIX have an interesting billing system, where they charge you a 95th percentile rate in 64Kbps increments. This in laymen's terms means that short bursts of internet traffic will account for a very small amount of average bandwidth usage over the period where we take the 95th percentile, the largest spikes are averaged out over the entire month. Ever since the inception of Telkom's volume billing on ADSL though, the other hosting providers have followed suit by allocating per GB rates to hosting. At this point, they killed the local hosting industry.

Let me explain why.

For an operation like ours, we require multiple bare-metal servers running Linux. These need to be geographically separated because they provide global access. Unfortunately a single strand of SAT-3 fibre is a high risk to our global availability. The US poses the least risk, so that is where we host. There is another factor to consider though, and that is falling victim to DDoS attacks or sudden increases in high volume consumption. Because these servers are on incredibly large pipes (the smallest being 100Mbps and the largest being 10Gbps) monitoring bandwidth on our end becomes infeasible, and attempting to control it becomes impossible (read: too difficult to bother). However, a sudden surge of usage can consume the entire connection without us noticing any problems since you have enough bandwidth to handle it. When you add a volume price tag without controlling the physical bandwidth, you're left in a dangerous position.

Take Hetzner for example. Their base volume allocation is 5GB with an over-use charge of R90 per GB. That sounds all very well and good, but what is 5GB? Put simply at 1Mbps you would require 11 hours to transfer 5GB of data. Except you don't have 1Mbps of bandwidth, you have datacenter bandwidth which can peak at 100Mbps. That means it would take a mere 7 minutes to transfer 5GB of data. In fact, while you're fast asleep in your bed a determined attacker could give you a bandwidth volume bill of 351GB, or R31,590.00.
The only options Hetzner would likely provide you is to pay the amount, or shut down your server and go out of business.

Compare this to the usual volume provided on US servers which is 1200GB, and usually they will simply moan a little and tell you to be more careful if you go over instead of billing you. In fact, most of the US hosting providers will not bill you at all for overuse unless it is consistent, and then they will usually tell you to purchase a bigger monthly amount rather than charging a nasty per unit cost. Their units are also substantially larger.

The usual response to this from the current local hosting providers is "but bandwidth is expensive". Sure, bandwidth is expensive, but when it comes to large datacenter interconnects it is not billed to them by volume, it is billed by bandwidth. They make their money overselling that bandwidth on the basis that no one should be using 100Mbps all the time, and certainly few people will. That doesn't really explain the massive cost they are adding though.

IS sell their ADSL bandwidth at a cost of about R60 per GB of volume. In reality it is even less than that to their larger partners. This bandwidth must incur the cost of IPC from Telkom DSLAMs to their POP, local interconnect from their POP to their international routers and from there on SAT-3 or satellite connection to overseas. This bandwidth is downstream for them, since the most of this bandwidth flows from somewhere to their customers. A datacenter is the opposite of this, most of their bandwidth flows from them to some ISP customers. The ISP usually has an excess of upstream bandwidth they must sell - this goes to hosting providers, and usually for less. Somehow we have either inverted this situation and the content providers are subsidising ADSL access, or there is something extremely ugly going on with the hosting providers. How can a system which requires at least two separate leased line costs be cheaper than a hosting provider who requires only one?

Aside from this obviously broken pricing scam, there is no benefit for me or anyone to host in South Africa. In fact, the only reason they might is for improved latency to their target customers. Given the ridiculous wave of downtimes and peering problems seen at Hetzner and other places, I have to wonder if it's ever worth it even for the sake of latency.

You'd be forgiven for assuming the only reason power could appear to go out these days is because of load shedding, but you could be horribly wrong. Just like we very nearly were this weekend on Friday night (09/05/2008) when I received an alert message saying that the big UPS feeding a bunch of servers in our office had gone onto battery.

We had been having some issues with the power that day and attempted to log a call with Durban Metro - their exchange simply told us their call volumes were high so we assumed a wide spread issue. The power started fluctuating at about 10am, and was bouncing around between 230V and 260V. This wasn't bad enough that our equipment and protection measures couldn't cope, and later in the day it settled back down to 230V and remained there. We thought all was well. Surely Durban Electricity would have their own monitoring measures, I mean if our small operation spent a few grand on it how hard can it be? A tragic mistake to trust your government I guess...

At 20h48 an On Battery notification came through to me, remembering the issues during the day I decided it was best to check it out. I also recalled that we had no load shedding scheduled for that area, and it's a business area that seldom gets load shed in the first place - least without warning. It was also a clear night and there was no chance of a lightning strike being the cause.

I logged into the UPS monitoring server and checked how much time was left in the batteries, and noticed something else.

input.voltage: 308.1
input.voltage.maximum: 308.1
input.voltage.minimum: 307.8

This wasn't an error, and it was climbing higher. Considering we're supposed to be supplied 220V, this was a serious problem.

I got in my car and took a drive down to Bulwer Park where our offices are located in Glenwood. Nothing looked suspicious and it just looked like a power outage. A Durban electricity van was cruising around one of the side streets, but it drove away as I approached it. Soon after that a fire truck appeared near the substation, but no one got out and it also drove away. One would be forgiven for thinking the situation was well under control. I sat around for a few more minutes wondering if I should pester anyone at 9PM on a Friday night. A street lamp flickered brightly above me, and a shocking reality dawned on me "This power is still live!". I bolted to the office entrance and got our technical director on the phone. He suggested I go in and pull the breaker immediately, which I was well on my way of doing. After I pulled the breaker I checked my gtalk where the other person monitoring the UPS remotely said the voltage had dropped to 0 now from 319V.

The TD arrived with the electrician on the phone, and we began looking around. The workshop area was full of smoke, and probably would have been full of fire if we left it minutes longer, light bulbs had burst from their sockets and we knew we were in serious trouble.

A quick call to Durban Electricity and they said they would send someone. How could they not know already? They didn't send anyone. We waited a while, and got some testing equipment. We measured the voltage at the breaker and found it was now 404V on the red phase (measured both to neutral and to ground) and 398V on the blue phase, the yellow phase had a fluctuating voltage which seemed to be nothing. I phoned Durban Electricity again and warned them that if they don't put their arse into gear in the next few minutes, people could die. Only then did they cut the power.

We had to rush to clean things up and repair anything we could and unfortunately very few pictures were taken (dead units are rather boring and we can't open them if we want to send them in anyway).
The toll at this point was four aircons, a cisco 877 left on on my desk, two UPS's, two monitors and a computer which had caught fire.

The pictures of the PC's power supply are the most telling


The lies

Durban Electricity told other businesses the fault was caused by cable theft. This is a blatant lie. The power climbed during the day and over a long time, cable theft can't possibly cause this. It was clearly a transformer or power regulation failure.

Then it happened again.

At 21h00 exactly on Sunday (11th of May 2008) we had another on battery alert. The power had again climbed well over 400V, the breakers were pulled and thankfully most things were now off from the last over-voltage. This time I wasn't around, but the people who responded were able to catch a Durban Electricity person who said the problem was because fuses were stolen from the substation. He found a hole in the fence and the doors were never even locked to the transformer room. How can they have the same fault only 48 hours apart and not take any measures to prevent it?
None of this reasoning adds up though. It seems very difficult and rather purposeless to steal the fuses, but if it was the reason for the first over-voltage why did they not do anything to prevent it happening again? It also should not lead to an uncontrolled over-voltage because there should be protection measures in place, and this would also mean that an ordinary fuse blowing could also lead to an over-voltage. Many have suggested theft of the neutral fuse which would have caused over-voltage in star connection, but if that was the case how were we able to read the over-voltage from phase to neutral?

I'm just not buying it, and either way they tell their side of the story it still seems to boil down to the fact that there aren't enough safety measures, the response times to the faults were dismal and little care was taken to ensure our safety resulting in thousands of rands worth of damage and potentially a major fire hazard. Even when the line technicians knew about the issue, they took ages to disconnect the supply which was already at a dangerously high level and climbing.

I'll continue trying to find out more, for now I'm glad we're back on line and able to work, despite being a little hot thanks to the four air-conditioning units which have blown.