Posted by & filed under JavaScript, Node.js.

I gave a talk this morning on the JavaScript Event Loop at Penguicon 2013. Even though I had used JavaScript for several years, I didn’t completely comprehend how the Event Loop works until a few months ago. When the opportunity came to present at Penguicon, I figured this was as good of a topic as any. You can download the presentation below (or view it in your browser), and I’ll throw all the individual slides and the gist of what I said about them on this page.

Download as a Keynote, Powerpoint, or HTML presentation.

Slide 1/14: Introduction

The JavaScript Event Loop: Introduction

Slide 2/14: Credibility

The JavaScript Event Loop: Thomas Hunter Credibility

I’ve been a web developer for a while, starting at some smaller mom and pop shops (not listed), to a couple fortune 50’s, before finally ending up at smaller and smaller (and quicker and more advanced) companies. For most of that time I was doing procedural PHP and MySQL programming, before eventually moving to mostly JavaScript (both frontend and backend).

I’m currently working with Packt to get a book on Backbone.js published (which is a frontend JavaScript framework for building Single Page Applications). Be sure to keep an eye out for it and purchase several copies, even if you don’t intend on reading them.

Slide 3/14: MultiThreaded

The JavaScript Event Loop: MultiThreaded

Let me first begin the presentation by talking about something mostly unrelated to JavaScript; MultiThreaded programming. If an application is built to be MultiThreaded, it will make use of several of your CPU cores simultaneously. This means it can do number crunching in different places at the same time and we refer to this as Concurrency. An application built in this manner can be a single process within the Operating System. The Operating System itself usually gets to choose which cores an application will run on (even which core a single threaded application will run on).

One way to fake MultiThreaded-ness in SingleThreaded languages is to simply run several different processes and have them communicate with each other.

For the longest time, CPUs were getting faster and faster, but then Moore’s Law caught up, and we sorta hit a wall with how fast our CPUs can get. So, to make hardware faster, we now throw more CPU cores at the computer. In order to truly scale and use the hardware to its fullest, one needs to build applications which make use of all CPU cores.

MultiThreading isn’t all butterflies and puppy tails though. There can be some big issues with this type of code, particularly Deadlocks and Race Conditions. One such example of these kinds of issues is that if an application is running on two separate threads, both threads reads a variable from memory at the same time, and both attempt to update the value by adding 2 to it. If the existing value is 10, and thread A adds 2, it does so by writing 12 to the memory location. If thread B also wants to add 2, it still thinks the value is 10, and writes 12. The programmer would expect it to be 14 and ends up with 12, and there are no errors. This type of bug can be very hard to track down, and the worst part is that it will happen in an unpredictable way.

Slide 4/14: SingleThreaded

The JavaScript Event Loop: SingleThreaded

Now that you know what MultiThreaded means, lets talk about how JavaScript is not MultiThreaded. A JavaScript engine exists in a single OS process, and consumes a single thread. This means that when your application is running, CPU execution is never performed in parallel. By running the JavaScript engine in this method, it is impossible for users to get the Deadlocks and Race Conditions which plague MultiThreaded applications.

Developers often refer to their callbacks running in an unexpected order as a Race Condition, however it is not the same thing that happens to MultiThreaded applications, and can usually be solved and tracked down easily enough (e.g., use another callback).

Slide 5/14: Implementation

The JavaScript Event Loop: Implementation

There are three important features of a JavaScript engine that deserve mention. These are the Stack, the Heap, and the Queue. Now, different browsers have different JavaScript engines (e.g. Chrome has V8, Firefox has OdinMonkey, and IE has something written in BASIC called Chakra (just kidding!)) and each browser will implement these features differently, but this explanation should work for all of them.

Heap: The simplest part of this is the Heap. This is a bunch of memory where your objects live (e.g. variables and functions and all those things you instantiate). In the presentation I refer to this as Chaotic, only because the order doesn’t really matter and there’s no guarantee with how they will live. In this heap, different browsers will perform different optimizations, e.g., if an object is duplicated many times, it may only exist in memory once, until a change needs to happen, at which point the object is copied.

Stack: This is where the currently running functions get added. If function A() runs function B(), well you’re two levels deep in the stack. Each time one of these functions is added to the stack, it is called a frame. These frames contain pointers to the functions in the heap, as well as the objects available to the function depending on its current scope, and of course the arguments to the function itself. Different JavaScript engines likely have different maximum stack sizes, and unless you have a runaway recursive function, you’ve probably never hit this limit. Once a function call is complete, it gets removed from the stack. Once the stack is empty, we’re ready for the next item in the Queue.

Queue: This is where function calls which are queued up for the future go. If you perform a setTimeout(function() { console.log('hi'); }, 10);, that anonymous function is living in the next available queue slot. No items in the queue will be run until the current stack is complete. So, if you have some work that might be slow that you want to run after you get your data, try a setTimeout() with a delay of 0ms. Future items which rely on I/O to complete, or a long running timer, are somehow in that queue as well, although I’m not exactly sure how that is implemented.

It’s worth mentioning Garbage Collection here as well. In JavaScript it’s easy to create tons of objects all willy nilly like. These get added to the Heap. But, once there is no scope remaining that needs those objects, it’s safe to throw them away. JavaScript can keep an eye on the current stack and the items in the Queue, and see what objects in the Heap are being pointed to. If an object no longer has pointers to it, it is safe to assume that object can be thrown away. If you aren’t careful with how you manage your code, it’s easy to not have those pointers disappear, and we call this wasted memory a Memory Leak.

Slide 6/14: Implementation Example

The JavaScript Event Loop: Implementation Example

This code-run is an example of the previous slide. So, the very first thing that happens is that function a() and b() are “hoisted” to the top of the script, and are added to the heap. We then run the first message log “Adding code to the queue” in the current stack. After that we run a setTimeout, and the anonymous function in there is added to the Queue. Then we do another log, and run the a() function with an argument of 42. We are now one level deep in the stack, and that frame knows about the a() function, the b() function, and its argument of 42. Within a() we run b(), and we are now two levels deep in our stack. We print more messages, leave b(), leave a(), and print a final message. At that point, our stack is empty and we’ve run all of our code, and are now ready for the next item in the queue.

Once we’re in the next queue item, we run the anonymous function (which exists in the Heap somewhere), and display our message.

At first glance, one might assume the message “Running next code from queue” could have been run earlier, perhaps after the first message. If this were a MultiThreaded application, that message could have been run at any point in time, randomly placed between any of the outputted messages. But, since this is JavaScript, it is guaranteed to run after the current stack has completed.

Slide 7/14: Sleeping

The JavaScript Event Loop: Sleeping

I come from a background in writing PHP/MySQL applications. When a PHP script runs, it performs a bunch of work, and then probably runs a MySQL query. Once that call is made to the external server, the application falls asleep. It literally halts everything it is doing and waits for a response from the database server. Once the result comes back, it does some further processing, and then it might perform another I/O function, such as calling an RSS feed. And, as you might guess, it falls asleep again.

Now, what if the call to the RSS feed doesn’t require any of the data we gain from the database call? Then the order of the two calls might not have mattered. But, more importantly, the two calls could have been run simultaneously! The application is as slow as the two calls combined, instead of being as slow as the slowest of the two.

Node.js does something pretty cool, where every I/O request it makes is a non blocking call. This means that the call can end the current stack, and the callback can be called later on in a separate Queue. If we’re performing a bunch of I/O operations, they can be run in parallel. The application will still sleep, but it won’t be blocking.

The web browser is the same. Most of the time it is doing nothing, perhaps waiting for a user to click on something, or waiting for an AJAX request to finish up.

Slide 8/14: Sequential vs Parallel I/O

The JavaScript Event Loop: Sequential vs Parallel I/O

This is a great graphic I adapted from the CodeSchool Real-Time Web with Node.js course. It shows how the I/O operations for sequential I/O compares to parallel I/O. The sequential graph represents calls make in a more traditional language such as PHP, whereas the parallel graph represents calls made in an EventLoop driven language with non blocking I/O, or even MultiThreaded applications. Notice that the application is only as slow as the slowest I/O operation, instead of as slow as all I/O operations combined.

Slide 9/14: Other Language Event Loops

The JavaScript Event Loop: Other Language Event Loops

JavaScript isn’t the only language that can have an Event Loop. They can be implemented in the more traditional procedural languages as well. However, by having it built into the language, it’ll surely be quicker and have a nicer syntax.

Also, when it is implemented in another language, you lose out on the special features if your I/O is blocking, so you’ll have to be careful with which libraries you choose.

Some examples of Event Loops in other languages include Ruby’s EventMachine, Python’s Twisted and Tornado, and PHP’s ReactPHP.

Slide 10/14: Other Language Event Loop Example

The JavaScript Event Loop: Other Language Event Loop Example

Here’s an apples to oranges comparison of the Event Loop working in Node.js to perform a simple TCP echo example, and the (I’m assuming) same application working in Ruby’s EventMachine. I took the Node example from the homepage of nodejs.org, and the EventMachine example from their GitHub readme. They’ve been altered slightly to use the same text and hopefully perform the same function (I honestly don’t know Ruby though).

Notice that the syntax for JavaScript is less terse.

Slide 11/14: Event Loops are Awesome

The JavaScript Event Loop: Event Loops are Awesome

There you have it folks, Event Loops are awesome. They don’t have the race conditions or deadlock issues that MultiThreaded applications have. Most web applications waste time waiting on I/O, and this is a good way around it. There is no special syntax for it to work in JavaScript; it is built in. It’s pretty easy to build stateful web applications (whereas if this were PHP you’d need a database to store shared data, in JS you could just use a local variable).

Slide 12/14: Event Loops aren’t Awesome

The JavaScript Event Loop: Event Loops aren't Awesome

There you have it folks, Event Loops aren’t awesome. If you perform a bunch of CPU intensive work, it will block your process and only use one core. Unless, of course, you use Node.js and offload work to another process. Or, if you’re in a browser, read the next slide. Memory leaks are also possible, as you’re running an application for a long time instead of temporarily. Unless, of course, you program cleanly and are able to avoid those issues.

Slide 13/14: Web Workers

The JavaScript Event Loop: Web Workers

Well, now that I spent this whole time telling you how JavaScript is a SingleThreaded application and you can’t make use of multiple cores, I’ll apologize for being a liar. The core of JavaScript is single threaded, and it’s been that way for many years. However, there’s this cool new thing that came out in the last few years called Web Workers. It will allow your browser (doesn’t exist in Node) to offload work to a separate thread. This feature is available in every modern web browser, so feel free to offload your work today.

How it works is you create a script, and throw some specifically formatted code in there. The main script loads it with var worker = new Worker('task.js');, where task.js is an existing JavaScript file. You also attach a bunch of event handlers to the created worker object, and interact with the worker that way. The script will run in its own instance of the JavaScript engine, and cannot share memory with the main thread (which has the nice side effect of preventing those race conditions).

When you want to pass information to and from the worker, you use something called message passing. This allows you to pass simple JSON objects around, but not complex objects that contain functions or anything referencing the DOM. A great use-case for Web Workers would be calculating a SHA1 hash or performing some map/reduce computations. Basically, anything that involves a ton of number crunching and isn’t all DOM operations.

Slide 14/14: Conclusion

The JavaScript Event Loop: Conclusion

There you have it, the JavaScript Event Loop. It is great for I/O bound applications, and horrible for CPU bound applications. Many people think the engine is MultiThreaded, or at least that it can do things in parallel. Turns out it can do I/O in parallel, but not CPU computations (unless using a separate process with Node.js or a Web Worker in the browser).

Posted by & filed under Personal.

Growing up I had a slight case of Synesthesia and would strongly associate colors with numbers. Sevens were Yellow, Fours were Orange, and Eights were Black. Other numbers had colors too, but the association wasn’t as strong. I also only had colors for the different digits, I didn’t see big numbers as their own colors, but each digit of color.

Here’s an example of a bunch of numbers in the Fibonacci Sequence with my Synesthesia colors applied.

Posted by & filed under OS X.

Go to your Finder in OS X, right click on a file, and click the Open With submenu. Does it look something like this, what with all the duplicated applications and whatnot?

OS X Finder Open With Duplicate Entries

If so, congratulations on breaking your Finder!

Fixing it, however, is pretty easy. Open up your Terminal app, and run the following commands:

/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -kill -r -domain local -domain system -domain user
killall Finder

Once that is done, Finder will restart, and your disfigured context menu will magically be pretty again.

OS X Finder Open With Duplicate Entries Fixed

 

Posted by & filed under OS X.

OS X comes with a pretty handy tool for working with disks and burning disk images called Disk Utility. Unfortunately, it doesn’t seem to work for burning ISO images to USB drives. Chances are, if you’re reading this, you’ve already tried to burn an ISO image to a USB drive using the utility and were greeted with some sort of cryptic errors like these:

To restore the disk image, Disk Utility must scan it first.

To restore the disk image, Disk Utility must scan it first.

 

Unable to scan ISO file. (Invalid argument)

Unable to scan ISO file. (Invalid argument)

 

Basically, it means that Disk Utility has no idea what the hell it is doing.

Luckily for us, this is a BSD machine, and the terminal has some useful commands for us. Open up terminal, and run the command `diskutil list`. Assuming your USB drive is plugged in (hint hint), you will be greeted with a list of drives similar to this:

/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *128.0 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS System                  127.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *750.2 GB   disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                  Apple_HFS Media                   649.8 GB   disk1s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk1s3
   4:                  Apple_HFS Server                  99.3 GB    disk1s4
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *8.0 GB     disk2
   1:             Windows_FAT_32 BFPORTABLE2             8.0 GB     disk2s1

Of course your partitions won’t look as cool as mine, unless you’ve ripped out your optical drive and shoved in a second harddrive. Anyway, one of those devices should be your USB drive. In my case, it is /dev/disk2.

At this point, you’ll want to unmount the partition, although you don’t want to eject the drive. I’m not really sure what that means, but it has the effect of removing the partition from being mounted in the filesystem (e.g. disk2s1 being mounted as /Volumes/BFPORTABLE2), but the disk itself is still accessible as /dev/disk2. You can open Disk Utility, right click on the Partition (but not the physical device), and click unmount (not eject).

After this, you can execute the `dd` command, which is used for duplicating disk drive images. You’ll want to run the following command:

sudo dd if=~/Downloads/DISK-IMAGE.iso of=/dev/DRIVE-NAME

What it all looks like in my terminal. You can see that I ran the `diskutil list` command to find my partition, attempted to run the `dd` command without `sudo` and it failed, then I ran it with `sudo` again and it failed, so I went to Disk Utility, ejected it, and ran it again and it succeeded. The process will take a while before it completes copying to your device.

Burn ISO to USB

Using dd to burn an ISO to a USB drive on OS X

If you happen to be looking for a good USB drive, I would hands down recommend the Kingston SE9 32GB. It’s only $12 on Amazon. I’ve had mine for years, it’s solid as a rock, and the solid minimalist aluminum casing looks great. I’ve burned several different Linux distros on it by now without problem.

Posted by & filed under Uncategorized.

I love pretty things. The default TextMate 2 icon is not pretty. It’s a big ugly pink flower.

tm2-icon

All of the filetype icons are also pink flowers. Everything looks the same and it’s hard to know what files you’re looking at at first glance.

I downloaded a bunch of icons from Icon Archive, and went through replacing a bunch of images for the filetypes I commonly use. The result is this:

Screen Shot 2013-04-01 at 9.29.12 PM Screen Shot 2013-04-01 at 9.29.29 PM

Much better! Not perfect, but better. The icons don’t match in the slightest (they’re from several different authors) but at least you can identify at a glance what different filetypes you have and you don’t have to look at an ugly pink flower.

To set these icons like I have, do the following:

  • Right Click on the TextMate2.app file
  • Select Show Package Contents
  • Navigate to Contents/Resources
  • Download This Zip, extract the contents, and overwrite the files in that directory

 

Posted by & filed under Reviews.

I recently purchased the SNES Retro USB Super Nintendo Controller from Amazon for $10, $15 shipped (yeah yeah… I signed up for prime a few days later).

This thing is a broken piece of shit. Don’t buy it. The device advertised at Amazon is by a company called Tomee, however the one I received is by a company called Hydra. After reading product reviews, it would appear that this is a common issue when purchasing this product.

I would plug this thing into my different computers, and wait for something to happen. Anything. Nada. I would then tail the system logs to see if anything would happen. Nope. So, I filled out the form for Amazon and opted to send it back for a new one. But, it costs another $5 to ship it, not to mention I’d have to leave work to go to the post office. At this point I realized it was a wash, and figured I might as well get some benefit out of it; so I tore it apart.

When I opened it up, I half expected it to be hollow. However, it did indeed have a board in it.

The soldering on this thing is atrocious. There’s also a bunch of dried white powdery stuff on it, I wouldn’t be surprised if it has received water damage.

There appears to be a location for a resistor on the board, but it is missing. I’m not sure if it is common practice to omit a component like this, or if that is a sure sign the device was never completed.

I noticed that at the locations where the wires are soldered to the board, the black wire might have been soldered onto the wrong spot. I desoldered it and moved it to where the label was, but it was of no sure.

Anyway, here’s a bunch of pictures, and some of the text from the board in case anyone googles this (I couldn’t find anything online).

BL-SFC-USB-01
VER: 1.1
08/12/29
T=1.6mm
SNES USB Controller: Box

SNES USB Controller: Box

SNES USB Controller: Top

SNES USB Controller: Top

SNES USB Controller: Inside

SNES USB Controller: Inside

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Circuit Board

SNES USB Controller: Bad Wiring?

SNES USB Controller: Bad Wiring?

SNES USB Controller: Rewired

SNES USB Controller: Rewired

Discussion with Amazon

Discussion with Amazon

Posted by & filed under Reviews.

After a recent case of video game nostalgia and hunting down NES/SNES emulators, I wanted a game controller to play them on (instead of a computer keyboard). My first choice was a knockoff SNES controller with a USB port I grabbed on Amazon. Unfortunately, it didn’t work (product breakdown coming soon). So, I went out and grabbed the $10 ION GO PAD from from Amazon, having seen a friend with one IRL.

First Impression: It’s Small

My first impression with this thing; it is tiny! Here’s a couple of pictures I took for scale. Luckily though, while using this thing, the small size isn’t that big of a deal. I do have formidable man-hands, and don’t find it that bad.

ION GO PAD Hand ION GO PAD Scale

Works Everywhere, Plays Fine

I plugged this thing into my Mac for most of my game playing, it worked great out of the box. In fact, the first thing I did was throw away the little CD that came with it. It even worked while using ZSNES on my Ubuntu laptop. I have yet to try it on a Windows based computer, but if history is any indicator, this device will work just fine.

Playes some action games like Metal Slug using VisualBoyAdvance, worked great. I was able to maneuver my character around and die as often as expected. I then ran snes9x with Killer Instinct, able to play as expected. No problems with annoying accidental jumps, something that is common with cheaper gamepads and fighting games.

What Sucks

There are a few drawbacks to this thing. The button placement is a bit stupid. There are two buttons that you’ll never ever need to use, one is called Turbo and the other is Clear. Press Turbo then a different button, and that button now refires press events quickly. Clear can be used to disable that. Any good emulator is going to have this feature built in, not to mention it’s a dumb feature that nobody needs. Also, there’s no start and select button, which are common among many different consoles. Those Turbo and Clear buttons can’t be reassigned. Luckily, there are four shoulder buttons, and I find myself mapping the inner ones to select and start.

Also, it is possible to move right or left while holding the DPAD downward, if you put move your thumb rightward or leftward. Once you get used to the controller it’s pretty easy to prevent this, but it is pretty annoying at first.

The device folds open, and between the two folds is thin ribbon cable for connecting the two sides. I have a feeling that after opening and closing this thing a couple hundred times, one of those wires is going to get severed and the product will be useless.

Conclusion

I do recommend the ION GO PAD for anyone looking for a small, cheap USB game controller. Just toss into laptop bag and go. It’s really tiny, and easily fits into my laptop bag or a pocket. Keep in mind though that is a $10 gamepad, and isn’t going to be the best of quality.

If size and cost are not prohibitive, go grab something nicer like a $35 Wired Xbox 360 Controller. Anytime you’re dealing with a game controller, official console ones are always better than these dinky third party ones.

Posted by & filed under PHP, Security.

While looking into creating an OAuth 1.0a provider using PHP, I came across this article by Rasmus Lerdorf on using the PECL OAuth extension. He had some code in place for generating a consumer key and private key pair, and it would chop up a sha1 hash and use those, which works fine and provides plenty of randomness.

I’ve personally never been a big fan of passing a bunch of hexadecimal hashes around, as it involves a lot of wasted entropy. Since hashes like this can only convey 16 different characters, they aren’t that efficient. Something like Base64 is much nicer, as it has 64 possible characters, and in theory the same data could be described in one fourth the number of characters. It’s really just a personal choice though.

Anyway, I took Rasmus’s code, and modified it a bit. Instead of generating a SHA1 (40 hex chars) and hacking that up, I generate a SHA512 (64 hex chars), and base62 encode that (which is 0-9, a-z, A-Z, leaving out the / and + and trailing = characters of base64). This way, I still get extra entropy, and don’t have the hard-to-encode characters of base64.

The code looks like this, and one pair can be generated in about 0.0006 seconds.

function gen_oauth_creds() {
	// Get a whole bunch of random characters from the OS
	$fp = fopen('/dev/urandom','rb');
	$entropy = fread($fp, 32);
	fclose($fp);

	// Takes our binary entropy, and concatenates a string which represents the current time to the microsecond
	$entropy .= uniqid(mt_rand(), true);

	// Hash the binary entropy
	$hash = hash('sha512', $entropy);

	// Base62 Encode the hash, resulting in an 86 or 85 character string
	$hash = gmp_strval(gmp_init($hash, 16), 62);

	// Chop and send the first 80 characters back to the client
	return array(
		'consumer_key' => substr($hash, 0, 32),
		'shared_secret' => substr($hash, 32, 48)
	);
}

Here are some example pairs of Consumer Keys and Shared Secrets generated using this code:

lInXLgx6HbF9FFq1ZQN8iSEnhzO3JVuf : 6kvwVhDGUQGXFawmulAhvORRV8HpZy5OMqMVH7xqwkLcvTbo
uFKE99LWQ8PDdBFQiO6fUhRRLoagFcDT : RGa8MqS4NZtAbWM8Voloi9kSVPg5AoKc5kySZxoYtpkOFgHU
NZqgHW5oOkxHxc5OgVXkJGQ0Kus3rOjz : QQsfDh5Q7S1BB8UHiB0Ni3okKn8lEEbeDx1k4k2OjT2jWuzr

Let me know if you see any security issues with the above code, I admit that I don’t fully understand the gmp_ functions.

Posted by & filed under GIT, OS X.

Are you unhappy with the command line diff tools provided by git? If so, this tutorial is for you.

FileMerge is a utility that comes with OS X / Xcode, although it is hidden and not the easiest to get to. You can dig around the Xcode.app directory until you find it, although the command ‘opendiff’ should already exist in your terminal. To see a visual diff of two files, run ‘opendiff file1 file2′ in your terminal, and a GUI app will appear.

To set FileMerge as your git diff tool, first create a simple script that will be the translator between git and the FileMerge utility. I keep mine in the following location:

vim ~/bin/git-diff.sh

The contents of the file will look like this:

#!/bin/sh
/usr/bin/opendiff "$2" "$5" -merge "$1"

Once the script has been made, you’ll want it to be executable

chmod u+x ~/bin/git-diff.sh

Finally, tell git that you want to set it up as your default merge tool:

git config --global diff.external ~/bin/git-diff.sh

If you later decide you hate it, run this command to go back:

git config --global --unset diff.external