Last night I was faced with a daunting task of making a meta-heuristic process a non-trivial amount of data in no more than a tenth of the time it was taking now.

Just to help you embrace the herculean task this was: My goal was to make the algorithm’s runtime lower than 1.5 seconds for any realistic data sample. It was taking anywhere from 5 to 20 and even 30 seconds at the time.

So I set off. First some profiling and some tidbits here and there, then tweaking and poking and yanking and scraping code for hours upon hours upon hours.

Nothing, no increase in performance. Nothing even remote to sane behaviour at all. Even though the algorithm was working, my debugging/profiling data wasn’t making any sense.

And I said fuck this shit and went to chillax and rest my mind over Men Who Stare at Goats for two hours.

There were some good laughs and some pretty good chillaxing being done. My I-don’t-know-which-exactly-or-what’s-it-called part of the brain was probably busy processing the afore mentioned problem in the background. Or maybe it wasn’t, who knows.

All I know is that when I came back to the problem I noticed something funky.

HEY THAT FUNCTION ARGUMENT SHOULDN’T BE A FUCKING CONSTANT WHAT THE FUCK IS GOING ON!

Characters 1 and t look a lot alike …

So anyway, long story short. The meta-heuristic that used to take 1500 epochs to do something now takes only ten to do the same thing, but slightly better.

That’s a pretty fucking good improovement right there!

Now the algorithm is constantly taking about a second to do its thing and I’m happy. Pushed it into production where profiling data will be collected of a more varied sample of data to see if all is well.

Chillaxing! It really does work.

Tags: Algorithm, Brain, food for thought, life

This is a howto that might come in handy to some people, but mostly I just want to document how I poked around some very angry django dragons and created something marvelous. There are also people on twitter who were wondering what the fuck I was doing.

So let’s start by describing the problem. We have a base user model named pUser (yes stupid naming convention) that is tied to a cookie, which holds an id. These users are then tied to a number of different API accounts. In my case it is Delicious, Twitter and Facebook. The user_id is also used to tie a bunch of meta data in different other models to them.

The problem is that we do not want to trouble users with a special login for our service. But they are using different computers and browsers, so the same physical user can have multiple user id’s.

However through their Delicious et al. credentials we can tie them back together into a single entity. But we do not want to trouble the rest of the code with this detail, it should just work seamlessly because otherwise we’d be forced to introduce checking for this stuff at about 50 different places in the project.

My approach to solving this goes as follows; at the end will be the three tests that indicate that the solution is valid. A hardcore test through the actual UI also confirmed that everything works.

Funky geek stuff follows, you have been warned

First we introduce a model that connects different user id’s to the main user (i.e. the first id said user was given)

class UserNormalisation(models.Model):
	main_id = models.IntegerField()
	sub_id = models.IntegerField()
 
	class Meta:
		unique_together = ("main_id", "sub_id")

Then we give our Delicious model a ModelManager that will perform duplicity checking and tie different users together as needed.

class DeliciousManager(models.Manager):
	def create(self, **kwargs):
		try:
			old = Delicious.objects.get(username=kwargs['username'])
			new = super(DeliciousManager, self).create(**kwargs)
			try:
				UserNormalisation(main_id = old.user.id,
						  sub_id = new.user.id).save()
			except IntegrityError:
				pass
			new.delete()
			return old
		except Delicious.DoesNotExist:
			return super(DeliciousManager, self).create(**kwargs)
 
class Delicious(models.Model):
	user = models.ForeignKey( pUser )
	username = models.CharField( max_length=255 )
	password = models.CharField( max_length=255 )
	isScrobbled = models.BooleanField( default=False )
 
	objects = DeliciousManager()

Basically when the createfunction is called it checks whether a Delicious model with the same username already exists and if it does, then a row is added to the UserNormalisation table to tie the two user id’s together.

And here’s the real magic, the changes we did to the pUser model.

class pUserManager(models.Manager):
	def get(self, **kwargs):
		user = super(pUserManager, self).get(**kwargs)
		try:
			id = UserNormalisation.objects.get(sub_id=user.id).main_id
			user = super(pUserManager, self).get(id=id)
		except UserNormalisation.DoesNotExist:
			pass
		return user
 
class pUser(models.Model):
	username = models.CharField( max_length=50 )
	password = models.CharField( max_length=255 )
	creation = models.DateTimeField( auto_now=True )
 
	objects = pUserManager()
 
	def __init__(self, *args, **kwargs):
		super(pUser, self).__init__(*args, **kwargs)
		try:
			id = UserNormalisation.objects.get(sub_id=self.id).main_id
			self.id = id
 
		except UserNormalisation.DoesNotExist:
			pass

The pUserManager should have a few more functions that do essentially the same thing for other operations (filter comes to mind). Essentially whenever a pUser is fetched from the db the manager will return the real user as per the UserNormalisation model.

Another trick that makes this work seamlessly even when used as a connecting model (primary key for instance) in a different table is that __init__ function. What I’ve discovered is that there it’s enough to just change the user’s id in place and everything will work.

Here are the tests that confirm all of this funky stuff actually performs as expected

	def test_normalisation(self):
		user = pUser(username="test", password="test")
		user.save()
 
		user2 = pUser(username="test2", password="test")
		user2.save()
 
		norm = UserNormalisation(main_id=user.id, sub_id=user2.id)
		norm.save()
 
		fixture = pUser.objects.get(id=user2.id)
		self.assertEquals( fixture.id, user.id )
 
	def test_normalisation2(self):
		user = pUser()
		user.save()
		user2 = pUser()
		user2.save()
 
		user.delicious_set.create(username="test", password="test")
		fixture = user2.delicious_set.create(username="test", password="test")
 
		self.assertEquals( fixture.user.id, user.id )
		self.assertEquals( UserNormalisation.objects.get(sub_id=user2.id).main_id, user.id )
		self.assertEquals( fixture.user, user )
 
	def test_normalisation3(self):
		user = pUser()
		user.save()
		user2 = pUser()
		user2.save()
 
		user.delicious_set.create(username="test", password="test")
		fixture = user2.delicious_set.create(username="test", password="test")
 
		norm = UserNormalisation.objects.all()
 
		Concepts.relate(user=user2, concept1="tag1", concept2="tag2")
		relation = ConceptRelation.objects.filter(user=user2, concept1="tag1")[0]
		self.assertEquals( relation.user.id, user.id )
		self.assertEquals( relation.user, user )

Take special note to the latter two examples. In test_normalisation2 you can see that when a delicious_set is created for user2, the two users become the same thing because both we’re using the same delicious username both times. Something similar happens in test_normalisation3, but there we see that creating a ConceptRelation for user2 actually creates it for the first user because they both behave as if they were the original user.

Tags: django, food for thought, intrigues, Programming, python

Ah, finally, we can breathe again!

The last couple of weeks … was it two, was it three? Can’t really tell, feels like I’ve left the office just yesterday while at the same time seeming like it’s been years since I last set foot in the office to do any real work.

This my friends, this ejection from the stream of time, this rejection from life, this abortion from the normality of our daily lives, this horrible feeling of floating in time like naught is happening but what you can see on your immediate palm …

… this is what exam season does to you.

For me it is, luckily, already over. Some of my best buddies have another week of wartime ahead of them. Waring with evil warlords, waging battle with delusional teaching assistants. To them I wish good luck! Good luck I say!

To the rest of us I say, Gidday mate! Finally, finally we have come out of that Vietnam that is the few weeks of exams. Those stressful days when your future hangs precariously at a balance and your parents are wagging their eternal finger into your face “you better pass those exams mister lest we reject you from the family unit and make you get a real job”

But alas dear mother! Alas I say! Alas! For I have passed three out of five exams and that means I’ve got a 60% success rate and not even all of those were a weak mere pass! Nay! For I am victorious! Victoriously have I come out of that hell that is pure suffering.

Could’ve done better though. Crap.

PS: said season also fucks with your mind.

Tags: food for thought, life

Lately we’ve been hearing a lot, and I mean a bloody metric shitload, of rumours, speculation and other fun things about the soon-to-be-announced Apple tablet. You know how those Apple fanboys get, the first time hype started building about this thing was several years ago, but lately it’s been getting soooo pervasive you just knew it was the real deal this time.

Of course after all this hype the features I expected were:

• the casing is made of solid gold
• it can make me a sandwich
• it brings coffee
• it fits in my pocket
• it is very very useful
• it can wipe my arse after I take a dump
• it can fly me to the moon and back
• it works like The Guide mk.2 (If you don’t know what this means you should be ashamed of yourself)

So let’s see what Apple gave us:

• oversized iPhone that can’t make calls

Errr … what!? Seriously Apple? Seriously? This!? Really!? We’re doing this again!?

Ok look, I love Apple, hell I even want an iPhone. And I really believe the devices they create are marvelous pieces of technology that work very well. But this fucking piece of crap is the biggest technological let-down I have ever had the displeasure of seeing.

I mean seriously, what the fuck was Apple smoking when they designed this thing? They’re supposed to be this sooper innovative company performing feats of magic right before our eyes, but instead, they take all the old technology, add nothing of the new, and call it the next big awesomest thing.

Fuck off Apple. Call me when you start making useful and exciting stuff again.

At least with the Newton it was marvelous technology that was too far ahead of its time, the iPad is just boring, mundane and boring.

Oh and you can tell they know it’s boring and useless because it’s PRICED THE SAME AS THEIR OLD PRODUCT!

Tags: Apple, food for thought, Handhelds, intrigues, IPhone, life, Steve Jobs, Technology

Size matters!

But obviously that’s a very lame argument so let’s work it out a bit.

Everybody knows that when it comes to impregnating homo sapienatic females any and all but the weirdest of male homo sapienatic instruments will do. It can be wide, it can be small, it can be whatever. As long as it’s longer than 5 inches you’re good and obviously since you’re here and we have this thing called evolution and inherited traits, chances are your phallic instrument is longer than 5 inches mister.

And 5 inches is quite alright, I wouldn’t want a 5 inch knife in my back …

But homo sapiens are smart cookies and they needed something more out of sex. This thing called an endorphinal high, we like drugs maaan, because after all, if sex would hurt nobody would do it and then we’d all be fucked. Not to mention that’s no fun at all.

So most of the time when a homo sapiens female and a homo sapiens male get together it isn’t to make little hobbits, it’s to make fun and create natural occuring brain drugs. Woo.

This however introduces a problem, the lukewarm hole most females sport is very stretchy and the more fun it’s having the more it stretches. The damn thing does not in the least care about what you’re putting inside, it just wants more and more and more and it gets bigger and bigger. Then the lady stops getting pleasure because she can’t feel anything.

Hell, there’s very little friction left!

What do? Well obviously you put a bigger instrument into the hands of the master male homo sapiens so that he could pleasure the female better. Problem solved!

By the by, many animal species have tricks to make the female more … uhm … willing. Like cats have spikes on their instruments and ducks have a spiral penis! Wow, amazing, shocker!

Ok so no bitching about “Size doesn’t matter, it’s how you use it” Sod off with those, size, in humans, is about 80% of the equation when it comes to pleasurable activities of the sort.

Therein lies the problem. The more the woman wants to have fun, the bigger instrument she needs. But how can you tell who’s got a big instrument when the damn bastards keep it neatly tucked away in their garment?

Confidence!

The more confidence a male sports the bigger their schlong! The mechanism for how big schlong == big confidence is probably self-evident so let’s leave it at this. The reason why bigger schlong equals bigger confidence is that lasses need a heuristic for finding bigger pleasure.

Tags: Adult, advice, Biology, Evolution, food for thought, Human, intrigues, Male, Species

You know how it is said every programmer needs to learn how to do parallelisation and funky stuff on multi-core multi-processor beast of machines? And how such machines aren’t even really beasts these days, they’re our run of the mill desktop and portable computers.

This is the world we live in.

It’s getting worse by the hour!

Very soon the first thing a young programmer will hear out of a lecturer’s mouth will be “Thread-Safe“.

But there’s something we can do about that even today. First of all, we can kiss threading good bye. Sure it’s sweet and yes it sort of works. But ew! It’s like trying to make a marine corps do their job with everyone’s finger in someone else’s arse. That’s the problem with threads you see, they keep picking each other’s arses and noses and then nobody can do any work.

Multi processing to the rescue!

Running an algorithm in several processes is the only thing that makes it run on several processesors in parallel and it gives each process its own memory space and everyone is nicely contained in their own little world. But fuck, now you can’t exactly pick another process’s arse when you need to … like when eating through a common queue of tasks.

And then python’s multiprocessing module, library, thingy, whatever it’s called, comes into play.

It. Just. Makes. Everything. So. Fucking. Easy!

This weekend I was working on a scrobbler for Delicious. Basically this thing is supposed to go through a user’s Delicious history, scrape every website it finds, send the results to three different semantic API’s and build connections between the tags those API’s return and the ones the user used to tag the particular link.

Now obviously there’s a lot of downtime involved here for every iteration. You’re easily looking at 10 solid seconds of waiting per website. This means that scrobbling 838 pages (my stress test) would take about two and a half hours. With multiprocessing it took something like 20 minutes.

The beauty of this approach is that I’ve never ever ever done anything in parallel. And yet I could do funky things like worker pools, queues, semaphoring and a bunch of other stuff I’ve only heard of in fairy tales until now … in an hour.

So there you go, an investment of a few hours for learning from scratch and some tweaking to create a ten-fold increase in speed.

Tags: Application programming interface, food for thought, life, Multi-core, Multiprocessing, Parallel computing, Process, Programming, Thread

This is the second post in the series of How <x> changed my style where I shall talk about tools and events that had a significant impact on my style of doing things. If you happen to like this idea, I would be very happy if you could help it spread like wildfire, because it’s a form of pay-it-forward where we say Thanks for cool stuff.

A few weeks ago, or was it days, I forget, I stumbled upon a rather fascinating video about living to be a hundred years old and more. But not just sitting around and waiting to die kind of 100+, the kind where at 100 you’re still happily running around, building fences and doing stupid crap most westerners really wouldn’t expect you to do anymore.

Everything in there is fine and good, but the most important bottom line I picked up on was the importance of having an ikigai. It’s all wonderfully explained in the video so I won’t go into what ikigai is, I’d rather say a little bit about what it feels like to discover one’s ikigai. (it doesn’t matter whether you know what it’s called or not, I didn’t for months)

As late as just last Spring my life revolved around going to a school I hated, having a job that was alright but not quite that and occasionally getting together with my girlfriend. I was doing some nifty stuff in the evenings and late at night, but mostly it was just another burden no matter how much I loved it and felt like I believed in it.

Then something changed, I can’t exactly put my finger on it, but it changed. The symptom of this change was that I quit my job and devoted most of my attention to the side-project, which evolved far far far away from what it was back then.

And suddenly much was different. School suddenly seemed interesting and awesome, I started learning exciting new things every day for the first time since being a kid. Getting up in the morning was … well it’s still fucking hard, I’m just not a morning person … but staying up at night was painless as hell. Meh, I won’t pretend like I can describe this, just try it.

Tags: food for thought, life, Pay it forward

Last night I watched a bloody amazing video of a university professor who was just oozing awesome. Seriously, this guy is all kinds of cool and I would love to someday have a professor that structures his lectures and classes like this guy does.

Now watch the video, I’m not here to blab, I’m here to make you watch this

Tags: Education, food for thought, Professor

This is the first post in the series of How <x> changed my style where I shall talk about tools and events that had a significant impact on my style of doing things. If you happen to like this idea, I would be very happy if you could help it spread like wildfire, because it’s a form of pay-it-forward where we say Thanks for cool stuff.

Some months ago, fuck has it been two? three? four?,  I attended a series of lectures on Lisp and functional programming and some other useful thingies by Simon Belak. At first it all felt like just another useful tool under my belt. Cool, Lisp, yeah, so what do I do with this? Meh, it’s cool, knowing this can’t do any more harm than a parenthesis thrown at a boomerangual trajectory smashing the windscreen of our hapless programmer.

However it did do damage, dear god it did so much damage! It fucked with my mind man, it fucked with everything. Nothing has been the same since!

No not really, but lately I have started noticing some pretty remarkable changes in the way I write code and more importantly, the way I think about code. Suddenly everything is a function is a function is a function! It’s quite remarkable really, sure I’ll often still code the very obvious object as an object, but I’m no longer forcing object oriented programming where it doesn’t belong.

And then there are even more radical examples of my brain changing, for the better I believe. Let’s take a simple comparison of writing a function that calculates the average of a list of numbers. In the old days I’d write it like so:

avg = 0
for key in self.tags:
	avg += self.tags[key]
avg /= len(self.tags)

These days, and it was fucking surprising when I noticed myself doing this, the same function gets written with a much bigger lisp:

avg = reduce(lambda a,b: a+b, tags.itervalues())/len(tags)

I really couldn’t go so far as to say whether this change in style and thinking is good or bad, but it certainly is interesting to just sit back and watch yourself from a distance as you morph and change before your very eyes.

So what’s changed your style lately?

Tags: Computer programming, food for thought, Functional programming, Languages, life, Lisp