Swizec Teller - a geek with a hatswizec.com

    Processing as-you-read in clojure

    • Six hard disk drives with cases opened showing...

      Image via Wikipedia

    Sometimes we're given a stupid algorithm or two to implement at this or that course at uni. This time 'round they wanted us to implement something called external matrix transposition ... the idea is that you have to transpose a matrix live from the hard drive because it is potentially so bloody huge you can't store a single row in memory.

    Which is a bit silly, who's got 60+ gig huge matrixes these days? You're doing it wrong if you need that many dimensions! VERY wrong!

    But anyway, what really struck me as a bit silly is that they were big nazis when it comes to how we're supposed to do this. Needs to compile with their specific command, can only be done in Java or C/C++ ... yeah fuck you and fuck your grade, let's learn something.

    So I went and did it in Clojure ... well not really since I didn't finish, but I came close.

    Surprisingly, it is in fact very very hard to process data live as it's being read in a functional environment. Every library that's already out there open-read-closes the file, at best you get something that can read lines and allows you to work on every line as it's being read, but oh no, not characters. Nope, what sort of bloody idiot would want that?

    Well ... me.

    So here is my implementation, probably not the most elegant thing out there since I'm new to this, of what Java people call "nextInt". It returns the next integer in a file as a proper integer and so on.

    (ns io (:require [clojure.contrib.io :as io]))
    (defn read-char [rdr]
    (char (.read rdr)))
    (defn delimiter? [c]
    (if (or (< (int c) (int '\0)) (> (int c) (int '\9)) (= c nil)) true false))
    (defn digit [c place]
    (* (- (int c) 48) (Math/pow 10 (- place 1))))
    (defn numify [seq acc]
    (if (= seq nil)
    (int acc)
    (numify (next seq)
    (+ acc (digit (first seq) (count seq))))))
    (defn read-number [rdr acc]
    (let [c (read-char rdr)]
    (if (delimiter? c)
    (numify acc 0)
    (read-number rdr (concat acc [c])))))

    You call it with an open reader and an empty [] thingy because, the latter because I couldn't figure out optional function arguments.

    And if anyone's interested, here's the rest of the code that opens a file where the first number denotes the number of columns in the matrix and then contains the rest of it delimited with spaces or tabs or anything non-numerical, and copies every row into a separate file so a transposition can then be done (didn't quite get to this part)

    (defn copy-number [rdr writer]
    (.write writer (str (read-number rdr []) " ")))
    (defn copy-row [rdr writer i]
    (copy-number rdr writer)
    (if (> i 1) (copy-row rdr writer (dec i))))
    (defn break-row [rdr i columns]
    (with-open [writer (io/writer (str "podatki-0-" i))]
    (copy-row rdr writer columns))
    (break-row rdr (inc i) columns)
    (catch Exception e
    (defn transpose-matrix [file-name]
    (with-open [rdr (io/reader file-name)]
    (let [columns (read-number rdr [])]
    (break-row rdr 0 columns))))
    (transpose-matrix "/home/swizec/Documents/APS2/input.txt")

    PS: that try/catch in there is fugly, but I didn't know how else to check for EOF *blush*

    Reblog this post [with Zemanta]

    Did you enjoy this article?

    Published on April 12th, 2010 in Clojure, Function, Java, Languages, life, Programming, Uncategorized

    Learned something new?
    Want to become an expert?

    Here's how it works 👇

    Leave your email and I'll send you thoughtfully written emails every week about React, JavaScript, and your career. Lessons learned over 20 years in the industry working with companies ranging from tiny startups to Fortune5 behemoths.

    Join Swizec's Newsletter

    And get thoughtful letters 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.

    "Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"

    ~ Ashish Kumar

    Join over 14,000 engineers just like you already improving their careers with my letters, workshops, courses, and talks. ✌️

    Have a burning question that you think I can answer? I don't have all of the answers, but I have some! Hit me up on twitter or book a 30min ama for in-depth help.

    Ready to Stop copy pasting D3 examples and create data visualizations of your own?  Learn how to build scalable dataviz components your whole team can understand with React for Data Visualization

    Curious about Serverless and the modern backend? Check out Serverless Handbook, modern backend for the frontend engineer.

    Ready to learn how it all fits together and build a modern webapp from scratch? Learn how to launch a webapp and make your first 💰 on the side with ServerlessReact.Dev

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️

    Created by Swizec with ❤️