When I started coding for our latest project at Preona a bit of an epiphany happened. I suddenly got django. Every pattern that used to feel a bit strange and I may have fought a little, suddenly became obvious and simple. Suddenly out of the blue my whole codebase is so marvelously organised and separated into files it brings a tear to my eye.

In the old days the average Django App in my project had a structure a file structure a bit like so:

• models.py
• views.py
• tests.py
• RandomActualWorkLogic.py
• AnotherRandomOfWork.py
• …

Sure this works quite alright, but it feels a little weird and after way too much work you suddenly realise that instead of the clean design you envisioned as you started out, you’re left with a crow’s nest of interdependancy, circular crappery and your project is in pretty miserable shape.

This point was really driven home when I figured two different django projects should share some code. Extrapolating a clean interface and making stuff work … was a nightmare. I actually gave up after a week and decided to start afresh.

And boy, was that the best decision ever!

Better application structure

The first step was to take up a better way to organise my codebase inside applications. It goes something like this:

• models.py
• managers.py
• views.py
• tests.py
• forms.py
• decorators.py
• processing.py — (this is for certain special views)
• urls.py
• settings.py

The first result of this organisation is that now everything has its own place. There are no StrangeWorker.py crappy things anymore. All of those reside in a special ‘application’ just for those called a lib. Whenever you are making something and are looking for a place to put it -> the file is an obvious pick!

Another great result is that it’s now much easier to keep applications contained within themselves. Because both its specific url config and its specific settings config are contained within the application you can easily switch it around between projects or even make it a standalone thingy that projects can simply include.

Also because the file structure makes sense now and all those WeirdoWorker.py are separated elsewhere as proper libraries, there is less of a temptation to create mixed and confusing dependancies all over the place.

Conclusion

That’s it for django application structure. If you’ve got any better ideas do mention them on twitter or via the comments below. If not, that’s alright

Either way, join me in the next installment of Django Protip when I’ll be talking about using “Forms, forms for everything”

Tags: Application programming interface, django, Languages, Programming, python, Web Frameworks

Sometimes we’re given a stupid algorithm or two to implement at this or that course at uni. This time ’round they wanted us to implement something called external matrix transposition … the idea is that you have to transpose a matrix live from the hard drive because it is potentially so bloody huge you can’t store a single row in memory.

Which is a bit silly, who’s got 60+ gig huge matrixes these days? You’re doing it wrong if you need that many dimensions! VERY wrong!

But anyway, what really struck me as a bit silly is that they were big nazis when it comes to how we’re supposed to do this. Needs to compile with their specific command, can only be done in Java or C/C++ … yeah fuck you and fuck your grade, let’s learn something.

So I went and did it in Clojure … well not really since I didn’t finish, but I came close.

Surprisingly, it is in fact very very hard to process data live as it’s being read in a functional environment. Every library that’s already out there open-read-closes the file, at best you get something that can read lines and allows you to work on every line as it’s being read, but oh no, not characters. Nope, what sort of bloody idiot would want that?

Well … me.

So here is my implementation, probably not the most elegant thing out there since I’m new to this, of what Java people call “nextInt”. It returns the next integer in a file as a proper integer and so on.

(ns io (:require [clojure.contrib.io :as io]))
 
(defn read-char [rdr]
   (char (.read rdr)))
 
(defn delimiter? [c]
  (if (or (< (int c) (int '\0)) (> (int c) (int '\9)) (= c nil)) true false))
 
(defn digit [c place]
  (* (- (int c) 48) (Math/pow 10 (- place 1))))
 
(defn numify [seq acc]
  (if (= seq nil)
    (int acc)
    (numify (next seq)
	    (+ acc (digit (first seq) (count seq))))))
 
(defn read-number [rdr acc]
  (let [c (read-char rdr)]
    (if (delimiter? c)
      (numify acc 0)
      (read-number rdr (concat acc [c])))))

You call it with an open reader and an empty [] thingy because, the latter because I couldn’t figure out optional function arguments.

And if anyone’s interested, here’s the rest of the code that opens a file where the first number denotes the number of columns in the matrix and then contains the rest of it delimited with spaces or tabs or anything non-numerical, and copies every row into a separate file so a transposition can then be done (didn’t quite get to this part)

(defn copy-number [rdr writer]
  (.write writer (str (read-number rdr []) " ")))
 
(defn copy-row [rdr writer i]
  (copy-number rdr writer)
  (if (> i 1) (copy-row rdr writer (dec i))))
 
(defn break-row [rdr i columns]
  (try
   (with-open [writer (io/writer (str "podatki-0-" i))]
     (copy-row rdr writer columns))
   (break-row rdr (inc i) columns)
  (catch Exception e
    nil)))
 
(defn transpose-matrix [file-name]
  (with-open [rdr (io/reader file-name)]
    (let [columns (read-number rdr [])]
      (break-row rdr 0 columns))))
 
(transpose-matrix "/home/swizec/Documents/APS2/input.txt")

PS: that try/catch in there is fugly, but I didn’t know how else to check for EOF *blush*

Tags: Clojure, Function, Java, Languages, life, Programming

This is a howto that might come in handy to some people, but mostly I just want to document how I poked around some very angry django dragons and created something marvelous. There are also people on twitter who were wondering what the fuck I was doing.

So let’s start by describing the problem. We have a base user model named pUser (yes stupid naming convention) that is tied to a cookie, which holds an id. These users are then tied to a number of different API accounts. In my case it is Delicious, Twitter and Facebook. The user_id is also used to tie a bunch of meta data in different other models to them.

The problem is that we do not want to trouble users with a special login for our service. But they are using different computers and browsers, so the same physical user can have multiple user id’s.

However through their Delicious et al. credentials we can tie them back together into a single entity. But we do not want to trouble the rest of the code with this detail, it should just work seamlessly because otherwise we’d be forced to introduce checking for this stuff at about 50 different places in the project.

My approach to solving this goes as follows; at the end will be the three tests that indicate that the solution is valid. A hardcore test through the actual UI also confirmed that everything works.

Funky geek stuff follows, you have been warned

First we introduce a model that connects different user id’s to the main user (i.e. the first id said user was given)

class UserNormalisation(models.Model):
	main_id = models.IntegerField()
	sub_id = models.IntegerField()
 
	class Meta:
		unique_together = ("main_id", "sub_id")

Then we give our Delicious model a ModelManager that will perform duplicity checking and tie different users together as needed.

class DeliciousManager(models.Manager):
	def create(self, **kwargs):
		try:
			old = Delicious.objects.get(username=kwargs['username'])
			new = super(DeliciousManager, self).create(**kwargs)
			try:
				UserNormalisation(main_id = old.user.id,
						  sub_id = new.user.id).save()
			except IntegrityError:
				pass
			new.delete()
			return old
		except Delicious.DoesNotExist:
			return super(DeliciousManager, self).create(**kwargs)
 
class Delicious(models.Model):
	user = models.ForeignKey( pUser )
	username = models.CharField( max_length=255 )
	password = models.CharField( max_length=255 )
	isScrobbled = models.BooleanField( default=False )
 
	objects = DeliciousManager()

Basically when the createfunction is called it checks whether a Delicious model with the same username already exists and if it does, then a row is added to the UserNormalisation table to tie the two user id’s together.

And here’s the real magic, the changes we did to the pUser model.

class pUserManager(models.Manager):
	def get(self, **kwargs):
		user = super(pUserManager, self).get(**kwargs)
		try:
			id = UserNormalisation.objects.get(sub_id=user.id).main_id
			user = super(pUserManager, self).get(id=id)
		except UserNormalisation.DoesNotExist:
			pass
		return user
 
class pUser(models.Model):
	username = models.CharField( max_length=50 )
	password = models.CharField( max_length=255 )
	creation = models.DateTimeField( auto_now=True )
 
	objects = pUserManager()
 
	def __init__(self, *args, **kwargs):
		super(pUser, self).__init__(*args, **kwargs)
		try:
			id = UserNormalisation.objects.get(sub_id=self.id).main_id
			self.id = id
 
		except UserNormalisation.DoesNotExist:
			pass

The pUserManager should have a few more functions that do essentially the same thing for other operations (filter comes to mind). Essentially whenever a pUser is fetched from the db the manager will return the real user as per the UserNormalisation model.

Another trick that makes this work seamlessly even when used as a connecting model (primary key for instance) in a different table is that __init__ function. What I’ve discovered is that there it’s enough to just change the user’s id in place and everything will work.

Here are the tests that confirm all of this funky stuff actually performs as expected

	def test_normalisation(self):
		user = pUser(username="test", password="test")
		user.save()
 
		user2 = pUser(username="test2", password="test")
		user2.save()
 
		norm = UserNormalisation(main_id=user.id, sub_id=user2.id)
		norm.save()
 
		fixture = pUser.objects.get(id=user2.id)
		self.assertEquals( fixture.id, user.id )
 
	def test_normalisation2(self):
		user = pUser()
		user.save()
		user2 = pUser()
		user2.save()
 
		user.delicious_set.create(username="test", password="test")
		fixture = user2.delicious_set.create(username="test", password="test")
 
		self.assertEquals( fixture.user.id, user.id )
		self.assertEquals( UserNormalisation.objects.get(sub_id=user2.id).main_id, user.id )
		self.assertEquals( fixture.user, user )
 
	def test_normalisation3(self):
		user = pUser()
		user.save()
		user2 = pUser()
		user2.save()
 
		user.delicious_set.create(username="test", password="test")
		fixture = user2.delicious_set.create(username="test", password="test")
 
		norm = UserNormalisation.objects.all()
 
		Concepts.relate(user=user2, concept1="tag1", concept2="tag2")
		relation = ConceptRelation.objects.filter(user=user2, concept1="tag1")[0]
		self.assertEquals( relation.user.id, user.id )
		self.assertEquals( relation.user, user )

Take special note to the latter two examples. In test_normalisation2 you can see that when a delicious_set is created for user2, the two users become the same thing because both we’re using the same delicious username both times. Something similar happens in test_normalisation3, but there we see that creating a ConceptRelation for user2 actually creates it for the first user because they both behave as if they were the original user.

Tags: django, food for thought, intrigues, Programming, python

You know how it is said every programmer needs to learn how to do parallelisation and funky stuff on multi-core multi-processor beast of machines? And how such machines aren’t even really beasts these days, they’re our run of the mill desktop and portable computers.

This is the world we live in.

It’s getting worse by the hour!

Very soon the first thing a young programmer will hear out of a lecturer’s mouth will be “Thread-Safe“.

But there’s something we can do about that even today. First of all, we can kiss threading good bye. Sure it’s sweet and yes it sort of works. But ew! It’s like trying to make a marine corps do their job with everyone’s finger in someone else’s arse.

Tags: Application programming interface, food for thought, life, Multi-core, Multiprocessing, Parallel computing, Process, Programming, Thread

Today I spent five solid hours programming (so far) and didn’t write a single line of code, nay, not even a single character. All I did was think with the whiteboard.

Eventually covering about 15 square meters of whiteboard in squiggly lines and weird characters and all sorts of stuff that I probably shouldn’t explain in too great a detail publicly. Let it suffice to say that I believe to have solved the problem of translating machine vocabulary to personalised user vocabulary.

Related articles by Zemanta

• Why Coder Pay Isn’t Proportional To Productivity (developers.slashdot.org)
• What does programmer productivity look like? (marginalrevolution.com)
• Thinking in Code (juanitachronowski.com)

Tags: Computer programming, life, Programming

Here I am at Hekovnik, alone in the romantic ambient of spotlights, whiteboards and loud music that makes me just wanna code code code. Yep, it’s Iron Maiden, felt like it, and son, I am not disappoint.

Lounging on the sofa, with my trusty MacBook in lap, shiny logo greeting anyone daring to enter my bubble of deep concentration and flowing code masterpieces.

It’s time like these that I truly love what I do, I know us developers are a fussy bunch and like to get all up in arms over the littlest thing that upsets us. That we love nothing better than to nag about some app or another taking 10 fucking milliseconds too long to open! The travesty!

But fact remains, I love being programmer, despite stupid tests not passing, despite frequent choices not to use testing that end up biting us in the arse, despite always changing design specs, despite all the bugs and all the weird shit and that little typo that brings the whole system to its knees and takes a week to fix …

… heaven really is where your code lies, because code is an expression of one’s self, because code is art and no matter what anyone ever says, I will continue to see myself more as a poet than an engineer.

Tags: intrigues, life, Programmer, Programming

When I took a break from php three months ago I’d been up to my shoulders in the technology for about five years – I remember my first interactions with the language were figuring out whether I need to use .php3 or .php4 or just .php file extensions. But that’s not very interesting.

A more interesting tale to tell is perhaps how for the last few months I was growing increasingly frustrated with my job, my work environment, my skills toolkit and simply the whole LAMP web development stack. It was time to take a break … it was a lot later that I discovered a cool TEDtalk on the topic, watch:

But anyway, onwards to what I’d learned.

Lessons that can only be learned by staying the fuck away

This week my PHP hiatus saw the beginning of its end, I had to get a freelancing job to pay bills and other such nuissance – since PHP and Javascript seem to be what I’m most popular for in the community that’s the easiest thing to get a job for. And to be honest, I’m not really fluent enough with anything else to do it for real money.

So earlier this week I had a meeting with the team lead of the project to give me a short tour of their, dare I say it, very well designed framework, and I realised how much extra work needs to be done by the framework just because PHP is such an incredibly lame language.

1. PHP doesn’t have support for that thing where you can use objects right out of a function. So we need a getScalar, getString, getRows, getRow function instead of a lovely getValue, “”+getValue(), getRows() and getRows()[0] system. Why php, WHY don’t you support this?
2. PHP also can’t do inline arrays, objects and tuples. This is great for passing data around in a descriptive way and PHP fails completely. Why can’t I do a return { x: 0, y: 100 }? Or something like that? It’s silly that I’d have to create a whole new Vector class that doesn’t do anything other than hold two values.
3. Another thing PHP desperately needs are keyword arguments, or whatever they’re called. It should be possible to call a function that has default values and define specific arguments without having to define all the other arguments because the one I need to set happens to be last. Seriously, what if the defaults suddenly change? Do I go through all the code to pass the correct defaults for that one argument I actually need? This could be solved by passing an object around but what’s this? Objects are a pain to create? Right …

So I guess that makes for almost one thing I’ve learned per a month of abstinence. Not the best ratio by all means, but it’s silly to find that such a modern language used by so many people would be such an utter failure on many points others, not unlike Lisp, have solved half a century ago. What the flying fuck!?

Did I miss anything? What else does PHP do well and what does it do poorly?

Tags: food for thought, PHP, Programming

Some of you may remember my writing about how unit testing is anti-productive and those of you I have to disapoint, I still dont’ think it’s the best productivity tool out there.

BUT!

The past few days I had another go at unit testing while developing a “server” for gathering statistics on how people use Twitulater. This was mostly in preparation for making a “server” for syncing Twitulater users, more on that some other day, it was mostly a learning exercise.

What I learned was that:

1. Unit testing is no guarantee for quality (con)
2. Unit testing can be really cumbersome to use and more time can be spent finding a test than solving the problem. Like for example: How do you test an app that reads from sys.stdin? (con)
3. Unit testing only helps with bugs you’ve anticipated or found
   

   Tags: Programming, Software testing, Twitulater, unit test

Today I finally got my feet wet with something I’ve been meaning to try on for size ever since reading Clean Code, during my trip to Vienna last autumn; wow has it really been that long already, where’d winter and spring hide?

Anyway, so at about midnight last night I finally embarked on the process of coding up a new feature for Twitulater, or at least its server-side bits, after some prep work. Now since this is a rather mission critical system that I’d very much like to be somewhat reliable and rock-solid, unit testing seemed to be the way to go.

Downloading PHPUnit gave me some headaches, because ubuntu’s package managers fucked something up. Why is it I can’t install Pear if there are broken packages for something completely different? Then came the figuring out of how to actually do unit testing and I must say, it was incredibly simple. Of course I did have eight months’ time for the principles grokked from Clean Code to seep in, but still.

Mostly I was suprised to finally confirm that I have been doing test driven development (TDD for you acronym junkies) for a long while. Just in reverse. Whereas now, with automated unit tests, I first write a test, then the code, I used to write the code then test the hell out of it. With PHPUnit testing has become much less of a hassle – run a command, make sure everything works. Nice!

However, unit testing has one large drawback. It took me a solid six hours of coding to produce … nothing. I have in my possession now 200 lines of real code that doesn’t implement much other than making sure users exist and authenticating them, and a bunch of tests taking up a full … 170 lines of code. This is of course after some refactoring and whatnot, the original ratio was always that there were twice as many tests as there was actual code. Yummie.

Now I’m sure many of you will bash me for being such a dirty little blasphemer, but fuck it. Despite all the headaches saved with testing, despite making testing a lot simpler and quicker. It still took me SIX BLOODY HOURS to create some functionality that never used to take more than two hours to code and debug.

Tags: food for thought, Programming, testing, Twitulater