Swizec Teller - a geek with a hatswizec.com

Senior Mindset Book

Get promoted, earn a bigger salary, work for top companies

Senior Engineer Mindset cover
Learn more

    Livecoding Recap #44: Dipping my toes in AR.js

    This is a Livecoding Recap – an almost-weekly post about interesting things discovered while livecoding. Usually shorter than 500 words. Often with pictures. Livecoding happens almost every Sunday at 2pm PDT on multiple channels. You should follow My Youtube channel to catch me live.

    This Sunday, we dipped our toes in AR.js. We didn't get very far with anything real, but fun was had and things were learned.

    AR.js is an open source library by Jerome Etienne that promises to bring "Efficient Augmented Reality for the Web - 60fps on mobile!". And it did. Right after I updated my phone to a beta version of iOS11.

    Why iOS11?

    Because augmented reality requires access to your camera. To get access to your camera, AR.js uses WebRTC. iOS doesn't support WebRTC in any browser until iOS11, which is meant to come out in the next 3 months.

    You can get the beta version until it's ready. So I did. Now I can AR 😁 AR.js in the browser, ARKit in native. Gonna have to play with that, too.

    iOS11 looks great btw! My iPhone 5SE is a little small for the new fluffy design, but I can appreciate that things are easier to see. Does that mean I'm getting old? 🤔

    Here's what we did to get a basic demo running

    We used create-react-app to bootstrap an app. This proved more trouble than it was worth because AR.js doesn't work well with import or require() statements. There's been some work to modularize it, but it hasn't landed yet.

    To get around this, we imported AR.js and aframe as <script> tags in public/index.html. This works well enough, but discards all the optimizations Webpack can do for us.

    <!-- public/index.html -->
    <script src="https://aframe.io/releases/0.6.1/aframe.min.js"></script>
    <script src="https://jeromeetienne.github.io/AR.js/aframe/build/aframe-ar.js"></script>
    <script src="%PUBLIC_URL%/aframe-minecraft.min.js"></script>
    <title>React App</title>
    

    aframe gives us WebVR support. I think it creates special HTML elements we can use to build VR and AR scenes. Not sure if they count as web components or not.

    AR.js gives us… I'm still not sure where aframe ends and AR starts. But all the demo code we used comes from the AR.js project, so I'm sure it's doing a lot :)

    aframe-minecraft is a demo of a dancing minecraft figure that Jerome uses in some of his videos. We aaaalmost got it working.

    We render the scene in src/App.js using strange HTML elements I have never seen before.

    import { Scene, Entity } from 'aframe-react';
    
    class App extends Component {
        render() {
            return (
                <div class="App">
                    <scene artoolkit="{{sourceType:" 'webcam',="" trackingmethod:="" 'best'}}="">
    
                        <a-anchor hit-testing-enabled="true">
                            <a-entity minecraft="" minecraft-head-anim="yes" minecraft-body-anim="hiwave" material="opacity: 0.5">
                            <a-box position="0 0 0.5" material="opacity: 0.5;"></a-box>
                        </a-entity></a-anchor>
                        <a-camera-static preset="hiro">
                    </a-camera-static></scene>
                </div>
            );
        }
    }
    

    App is a React component that renders stuff. We'll add more functionality next Sunday.

    Inside the render function, we use a combination of aframe-react, which is a thing wrapper on Aframe, and custom HTML elements coming from, I guess, AR.js.

    <Scene> creates a new WebAR/WebVR scene. a-entity is some sort of AR entity, whatever that means. In this case, a waving Minecraft figure that is supposed to be colorful and fun, but is instead pure black.

    Now that I think of it, we're probably missing texture files 🤔

    a-box creates a white semi transparent box. For some reason, this was necessary to make the Minecraft figure visible. I don't know why… maybe something to do with those textures.

    a-camera-static renders a full screen webcam view using the "hiro" image as an AR marker.

    AR.js is a marker-based augmented reality engine, which means it needs a recognizable image to attach itself to the real world. This means you can't render your stuff on top of any random object the camera sees. You need a specific marker.

    Like this →

    AR js data images HIRO

    The experiment code is on Github.

    Here's what we learned about AR.js

    AR.js is great, but it's early days for WebAR and augmented reality on the web. The experience was hacky and cool.

    You need markers, which limits usability. We can potentially improve this with on-the-spot deep learning that turns recognized objects into markers on the fly.

    You need a desktop browser, which doesn't need AR because why would it. Are you going to move your laptop around to look at augmented reality? Prob not.

    You need either an Android phone or iOS11. In a few months, everybody's phone is going to support AR.js. This is huge.

    WebRTC requires https. This makes development annoying because localhost doesn't have https, so you have to deploy on a real server if you want to test on your phone.

    5/7 would hack again.

    Published on August 28th, 2017 in Livecoding, Technical

    Did you enjoy this article?

    Continue reading about Livecoding Recap #44: Dipping my toes in AR.js

    Semantically similar articles hand-picked by GPT-4

    Senior Mindset Book

    Get promoted, earn a bigger salary, work for top companies

    Learn more

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️

    Created by Swizec with ❤️