Skip to content
Swizec Teller - a geek with a hatswizec.com

Processing as-you-read in clojure

  • Six hard disk drives with cases opened showing...

    Image via Wikipedia

Sometimes we're given a stupid algorithm or two to implement at this or that course at uni. This time 'round they wanted us to implement something called external matrix transposition ... the idea is that you have to transpose a matrix live from the hard drive because it is potentially so bloody huge you can't store a single row in memory.

Which is a bit silly, who's got 60+ gig huge matrixes these days? You're doing it wrong if you need that many dimensions! VERY wrong!

But anyway, what really struck me as a bit silly is that they were big nazis when it comes to how we're supposed to do this. Needs to compile with their specific command, can only be done in Java or C/C++ ... yeah fuck you and fuck your grade, let's learn something.

So I went and did it in Clojure ... well not really since I didn't finish, but I came close.

Surprisingly, it is in fact very very hard to process data live as it's being read in a functional environment. Every library that's already out there open-read-closes the file, at best you get something that can read lines and allows you to work on every line as it's being read, but oh no, not characters. Nope, what sort of bloody idiot would want that?

Well ... me.

So here is my implementation, probably not the most elegant thing out there since I'm new to this, of what Java people call "nextInt". It returns the next integer in a file as a proper integer and so on.

(ns io (:require [clojure.contrib.io :as io]))
(defn read-char [rdr]
(char (.read rdr)))
(defn delimiter? [c]
(if (or (< (int c) (int '\0)) (> (int c) (int '\9)) (= c nil)) true false))
(defn digit [c place]
(* (- (int c) 48) (Math/pow 10 (- place 1))))
(defn numify [seq acc]
(if (= seq nil)
(int acc)
(numify (next seq)
(+ acc (digit (first seq) (count seq))))))
(defn read-number [rdr acc]
(let [c (read-char rdr)]
(if (delimiter? c)
(numify acc 0)
(read-number rdr (concat acc [c])))))

You call it with an open reader and an empty [] thingy because, the latter because I couldn't figure out optional function arguments.

And if anyone's interested, here's the rest of the code that opens a file where the first number denotes the number of columns in the matrix and then contains the rest of it delimited with spaces or tabs or anything non-numerical, and copies every row into a separate file so a transposition can then be done (didn't quite get to this part)

(defn copy-number [rdr writer]
(.write writer (str (read-number rdr []) " ")))
(defn copy-row [rdr writer i]
(copy-number rdr writer)
(if (> i 1) (copy-row rdr writer (dec i))))
(defn break-row [rdr i columns]
(try
(with-open [writer (io/writer (str "podatki-0-" i))]
(copy-row rdr writer columns))
(break-row rdr (inc i) columns)
(catch Exception e
nil)))
(defn transpose-matrix [file-name]
(with-open [rdr (io/reader file-name)]
(let [columns (read-number rdr [])]
(break-row rdr 0 columns))))
(transpose-matrix "/home/swizec/Documents/APS2/input.txt")

PS: that try/catch in there is fugly, but I didn't know how else to check for EOF *blush*

Reblog this post [with Zemanta]

Did you enjoy this article?

Published on April 12th, 2010 in Clojure, Function, Java, Languages, life, Programming, Uncategorized

Learned something new?
Want to become a high value JavaScript expert?

Here's how it works 👇

Leave your email and I'll send you an Interactive Modern JavaScript Cheatsheet 📖right away. After that you'll get thoughtfully written emails every week about React, JavaScript, and your career. Lessons learned over my 20 years in the industry working with companies ranging from tiny startups to Fortune5 behemoths.

Start with an interactive cheatsheet 📖

Then get thoughtful letters 💌 on mindsets, tactics, and technical skills for your career.

"Man, love your simple writing! Yours is the only email I open from marketers and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"

~ Ashish Kumar

Join over 10,000 engineers just like you already improving their careers with my letters, workshops, courses, and talks. ✌️

Have a burning question that you think I can answer? I don't have all of the answers, but I have some! Hit me up on twitter or book a 30min ama for in-depth help.

Ready to Stop copy pasting D3 examples and create data visualizations of your own?  Learn how to build scalable dataviz components your whole team can understand with React for Data Visualization

Curious about Serverless and the modern backend? Check out Serverless Handbook, modern backend for the frontend engineer.

Ready to learn how it all fits together and build a modern webapp from scratch? Learn how to launch a webapp and make your first 💰 on the side with ServerlessReact.Dev

Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️