Swizec Teller - a geek with a hatswizec.com

Senior Mindset Book

Get promoted, earn a bigger salary, work for top companies

Senior Engineer Mindset cover
Learn more

    Processing as-you-read in clojure

    Sometimes we're given a stupid algorithm or two to implement at this or that course at uni. This time 'round they wanted us to implement something called external matrix transposition ... the idea is that you have to transpose a matrix live from the hard drive because it is potentially so bloody huge you can't store a single row in memory.

    Which is a bit silly, who's got 60+ gig huge matrixes these days? You're doing it wrong if you need that many dimensions! VERY wrong!

    But anyway, what really struck me as a bit silly is that they were big nazis when it comes to how we're supposed to do this. Needs to compile with their specific command, can only be done in Java or C/C++ ... yeah fuck you and fuck your grade, let's learn something.

    So I went and did it in Clojure ... well not really since I didn't finish, but I came close.

    Surprisingly, it is in fact very very hard to process data live as it's being read in a functional environment. Every library that's already out there open-read-closes the file, at best you get something that can read lines and allows you to work on every line as it's being read, but oh no, not characters. Nope, what sort of bloody idiot would want that?

    Well ... me.

    So here is my implementation, probably not the most elegant thing out there since I'm new to this, of what Java people call "nextInt". It returns the next integer in a file as a proper integer and so on.

    (ns io (:require [clojure.contrib.io :as io]))
    (defn read-char [rdr]
       (char (.read rdr)))
    (defn delimiter? [c]
      (if (or (< (int c) (int '\0)) (> (int c) (int '\9)) (= c nil)) true false))
    (defn digit [c place]
      (* (- (int c) 48) (Math/pow 10 (- place 1))))
    (defn numify [seq acc]
      (if (= seq nil)
        (int acc)
        (numify (next seq)
    	    (+ acc (digit (first seq) (count seq))))))
    (defn read-number [rdr acc]
      (let [c (read-char rdr)]
        (if (delimiter? c)
          (numify acc 0)
          (read-number rdr (concat acc [c])))))

    You call it with an open reader and an empty [] thingy because, the latter because I couldn't figure out optional function arguments.

    And if anyone's interested, here's the rest of the code that opens a file where the first number denotes the number of columns in the matrix and then contains the rest of it delimited with spaces or tabs or anything non-numerical, and copies every row into a separate file so a transposition can then be done (didn't quite get to this part)

    (defn copy-number [rdr writer]
      (.write writer (str (read-number rdr []) " ")))
    (defn copy-row [rdr writer i]
      (copy-number rdr writer)
      (if (> i 1) (copy-row rdr writer (dec i))))
    (defn break-row [rdr i columns]
       (with-open [writer (io/writer (str "podatki-0-" i))]
         (copy-row rdr writer columns))
       (break-row rdr (inc i) columns)
      (catch Exception e
    (defn transpose-matrix [file-name]
      (with-open [rdr (io/reader file-name)]
        (let [columns (read-number rdr [])]
          (break-row rdr 0 columns))))
    (transpose-matrix "/home/swizec/Documents/APS2/input.txt")

    PS: that try/catch in there is fugly, but I didn't know how else to check for EOF *blush*

    Reblog this post [with Zemanta]
    Published on April 12th, 2010 in Clojure, Function, Java, Languages, life, Programming, Uncategorized

    Did you enjoy this article?

    Continue reading about Processing as-you-read in clojure

    Semantically similar articles hand-picked by GPT-4

    Senior Mindset Book

    Get promoted, earn a bigger salary, work for top companies

    Learn more

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️

    Created by Swizec with ❤️