This is a Livecoding Recap – an almost-weekly post about interesting things discovered while livecoding. Usually shorter than 500 words. Often with pictures. Livecoding happens almost every Sunday at 2pm PDT on multiple channels. You should follow My Youtube channel to catch me live.
This Sunday, we dipped our toes in AR.js. We didn't get very far with anything real, but fun was had and things were learned.
AR.js is an open source library by Jerome Etienne that promises to bring "Efficient Augmented Reality for the Web - 60fps on mobile!". And it did. Right after I updated my phone to a beta version of iOS11.
Why iOS11?
Because augmented reality requires access to your camera. To get access to your camera, AR.js uses WebRTC. iOS doesn't support WebRTC in any browser until iOS11, which is meant to come out in the next 3 months.
You can get the beta version until it's ready. So I did. Now I can AR 😁 AR.js in the browser, ARKit in native. Gonna have to play with that, too.
iOS11 looks great btw! My iPhone 5SE is a little small for the new fluffy design, but I can appreciate that things are easier to see. Does that mean I'm getting old? 🤔
Here's what we did to get a basic demo running
We used create-react-app to bootstrap an app. This proved more trouble than it was worth because AR.js doesn't work well with import or require() statements. There's been some work to modularize it, but it hasn't landed yet.
not that i know of, i do <script>. there is a npm contrib, but im not sure how it helps - https://t.co/cvZkRE3G94
— Jerome Etienne #ARjs (@jerome_etienne) August 27, 2017
To get around this, we imported AR.js and aframe as <script> tags in public/index.html. This works well enough, but discards all the optimizations Webpack can do for us.
<!-- public/index.html -->
<script src="https://aframe.io/releases/0.6.1/aframe.min.js"></script>
<script src="https://jeromeetienne.github.io/AR.js/aframe/build/aframe-ar.js"></script>
<script src="%PUBLIC_URL%/aframe-minecraft.min.js"></script>
<title>React App</title>
aframe gives us WebVR support. I think it creates special HTML elements we can use to build VR and AR scenes. Not sure if they count as web components or not.
AR.js gives us… I'm still not sure where aframe ends and AR starts. But all the demo code we used comes from the AR.js project, so I'm sure it's doing a lot :)
aframe-minecraft is a demo of a dancing minecraft figure that Jerome uses in some of his videos. We aaaalmost got it working.
We render the scene in src/App.js using strange HTML elements I have never seen before.
import { Scene, Entity } from 'aframe-react';
class App extends Component {
render() {
return (
<div class="App">
<scene artoolkit="{{sourceType:" 'webcam',="" trackingmethod:="" 'best'}}="">
<a-anchor hit-testing-enabled="true">
<a-entity minecraft="" minecraft-head-anim="yes" minecraft-body-anim="hiwave" material="opacity: 0.5">
<a-box position="0 0 0.5" material="opacity: 0.5;"></a-box>
</a-entity></a-anchor>
<a-camera-static preset="hiro">
</a-camera-static></scene>
</div>
);
}
}
App is a React component that renders stuff. We'll add more functionality next Sunday.
Inside the render function, we use a combination of aframe-react, which is a thing wrapper on Aframe, and custom HTML elements coming from, I guess, AR.js.
<Scene> creates a new WebAR/WebVR scene. a-entity is some sort of AR entity, whatever that means. In this case, a waving Minecraft figure that is supposed to be colorful and fun, but is instead pure black.
Now that I think of it, we're probably missing texture files 🤔
a-box creates a white semi transparent box. For some reason, this was necessary to make the Minecraft figure visible. I don't know why… maybe something to do with those textures.
a-camera-static renders a full screen webcam view using the "hiro" image as an AR marker.
AR.js is a marker-based augmented reality engine, which means it needs a recognizable image to attach itself to the real world. This means you can't render your stuff on top of any random object the camera sees. You need a specific marker.
Like this →
The experiment code is on Github.
Here's what we learned about AR.js
AR.js is great, but it's early days for WebAR and augmented reality on the web. The experience was hacky and cool.
You need markers, which limits usability. We can potentially improve this with on-the-spot deep learning that turns recognized objects into markers on the fly.
You need a desktop browser, which doesn't need AR because why would it. Are you going to move your laptop around to look at augmented reality? Prob not.
You need either an Android phone or iOS11. In a few months, everybody's phone is going to support AR.js. This is huge.
WebRTC requires https. This makes development annoying because localhost doesn't have https, so you have to deploy on a real server if you want to test on your phone.
5/7 would hack again.
Continue reading about Livecoding Recap #44: Dipping my toes in AR.js
Semantically similar articles hand-picked by GPT-4
- Livecoding Recap #45: Almost building a ballistic missile with React and AR.js
- Livecoding #37- Building a real life color picker
- Livecoding Recap #46: 3D is hard, WebAR defeats me
- Livecoding #35 - A New Site for reactd3js.com
- Livecoding #30: Dipping my toes in React Native
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️