Crystal

After joining Grabble, and getting to know the team, I started having pretty regular chats with one of their resident designers, Ruban Khalid. I like to think I can design as well as code, so when he started showing the latest tools of the trade he used to design and prototype, he had my complete attention. He showed me Sketch, a sort of Adobe Illustrator competitor which a lot of the tech world are using now for design because of it’s pure ease of use. After giving it a spin myself I could see the appeal, considering my go-to tool had previously been Gimp. What caught my eye more than anything though was a pretty sweet feature called mirror. Using the Sketch iOS app you could preview your designs in realtime on any iDevice.

Of course, the first thing I did was search Google Play for the Android version.

(Do you see Sketch Mirror? I don’t see Sketch mirror.)

I gave it a quick Google but all I could find were guides that used combinations of different apps to produce a sort of Frankenstein setup. As a sanity check, I made sure with Ruban that there wasn’t anything I was missing. He did mention an app called Mira, but it frankly pretty buggy when I gave it a shot and required an extra app installed on desktop to function over wifi (it didn’t support USB). It was at about this point that a nice energy saving lightbulb appeared above me head.

I’d make one myself.

Research

There was no way I was going to fling myself head-first into this project though if it wasn’t viable, so I began to test the water. Obviously the Sketch team had created some sort of communication protocol, so my plan was to reverse-engineer and eventually use it myself. I would use Ruban’s iPhone to mirror designs and watch the traffic between the device and desktop. The Sketch app lets you connect over USB or WiFi, but I’ve no idea how to intercept requests/packets over USB, so, given the two, I opted to reverse-engineer the WiFi connection.

My first step whenever intercepting requests is to use Charles Proxy. It sits between the device and the internet by sending all HTTP requests to the desktop machine running Charles. I thought I’d see a bunch of requests pop up on-screen, but got nothing. Fair enough. That means there aren’t any HTTP requests taking place, so I’d have to count on Wireshark to sniff packets. I love Wireshark but it can be a bit verbose at times, which is why I only resort to it after checking for HTTP requests.

After capturing some packet dumps I spent a while poring over the data, looking for patterns as to the format of the messages.

A lot of junk, but there were some gold nuggets hiding in plain sight

I had recorded the IP addresses of my host machine (192.168.1.97) and of the iPhone (192.168.1.184) and started looking for references to the two. Before too long I started noticing some MDNS messages. MDNS (sometimes called Bonjour) is a service discovery protocol that devices on a network can use to broadcast services and discover services being broadcast by other network clients. In the Wireshark logs, you can see lines (#36-38) where Ruban’s iPhone sends out MDNS queries, looking for services supporting _sketch-mirror.. Then further down (#43-44), you can see responses from my host machine as it replies, telling the network that it supports the Sketch service. Neither MDNS message is sent to specific client IPs - that’s the magic of MDNS. It broadcasts every message to all clients on the network. This is great for discovery, but can lead to MDNS being blocked on corporate networks where there can be hundreds of computers and the MDNS traffic starts to overload the network.

Further down the logs I began to see more interesting UDP traffic between my two favourite IPs. I followed the UDP stream and was faced with a moderately mangled piece of data, depicting the communication between the two devices. Throughout I saw references to NSKeyedArchiver and some PNG headers.

It’s almost like it’s encrypted, but it’s not. Just security through obscurity.

After seeing the PNG headers I thought I’d try to extract the PNG from the data. I read up on the whole PNG header file structure.

The first eight bytes of a PNG file always contain the following (decimal) values:

137 80 78 71 13 10 26 10

I located the header in an awesome hex app I use called Hex Fiend.

The four-byte IEND image trailer chunk type field contains the decimal values

73 69 78 68

and located the end trailer. I’ve skipped all the cursing and trial and error, but eventually I cut down the hex code to just the PNG file and had this!

Totally worth it…

As nice as a non-fully transferred PNG image is to look at, I realised this wasn’t really a viable solution. I wasn’t going to write some sort of algorithm that spliced the PNG bytes from these requests, and I didn’t even know the sort of request I’d need to send from my app to trigger a response like this with an image in it. I’d need to find another way.

I desperately had one last look around the Sketch mirror interface. Inside the Sketch app was a link to view the artboards (that’s what Sketch calls each design) in a browser. I hadn’t noticed it before but sure enough, I had an overview of all my artboards in front of me, that updated as I edited them within Sketch - just like the iOS app.

The browser-based Sketch mirror

I’m much better versed in web/JS development than iOS, so this time round I could start digging into the sourcecode. It didn’t take long to find references to websockets within the code and after inserting some breakpoints, I began to document the undocumented internal websocket API of Sketch mirror through observation. Now I knew I had something tangible, and I wouldn’t even need a Mac app! I’d had experience with websockets before on Android, so I was buzzing at this point.

At this point I was confident that I would finish and had a little look into adding USB support. After initially proclaiming to Ruban and all my friends that USB would be a simple affair, I began to retract my earlier statement. I thought with Android being such an open platform and USB host support being available on so many phones now that it would be simple, but no matter what I tried, I could not work out how to communicate between the host machine and Android. I tried opening low level sockets, and using USB Host API’s but nothing worked, so after a few days I abandoned it readily, happy to get on with the design of the app.

Design

After using the iOS app I decided to intentionally base the navigation of my app off of it. I felt that the cost of context switching for an average designer using both the Sketch mirror app for iOS and mine would be too high and that the familiar patterns would help guide them through my app naturally without any explanation. Besides, it was such a simple design, and I really admired it.

The naming and icon design were a sort of continual thought process throughout. I knew I wanted a name that was somehow related to Sketch and its icon - obviously I could make a play on words using “mirror”, but for whatever reason I decided to play with Sketch’s diamond logo. With the advice of Ruban at hand I made an inspiration board, by finding all sort of images from the web that I liked the look of and that fit the mental model I had in my head. I settled on the idea of using a crystal structure, but I still had colours and shapes to play with. I traced a few images, played with them a little and switched them up.

Part of my inspiration board used origami crystals to produce concepts

I went through few crystal-like iterations, but eventually became attached to the idea of a paper-like logo, made of folded card and inspired by material design. I Googled some origmai letters for ideas, but unfortunately found one I liked too much. I accidentally just blatantly copied it and fell in love with it before realising I couldn’t just rip off someone else’s work. So I iterated. I changed a few things, chose my favourites, and morphed them. Slowly, after many iterations and countless hours, the design had shifted from something that I found on the internet to my own design, complete with its own tiny details no one would ever see.

Not every change was a big one but the cumulative result of many changes was

Even during the app’s design process I began to implement my new unofficial Sketch API in a small skeleton of an Android app. It was completely bare-bones and simply performed a handshake with Sketch, connected and showed the first Artboard fullscreen. Close to useless, yes, but it was solid evidence which convinced me it was all possible, and gave me the dopamine hit to motivate me further. As I finished designing parts of the app I would transfer the designs across to Android layout XML but without any animations or small details. The only problem with this approach was its tendency to completely confuse me. More than once I found myself trying to connect to Sketch, and my phone refusing to respond. Only about a minute later would I realise I was actually already connected and trying to tap on an artboard mockup of the app, rather than the real thing. True Sketch-ception.

You tell me. Are you looking at the design or the real app?

Around the 10 day mark I began to finalise it all. After adding small touches like sorting available servers by your most frequently connected and double tapping artboards to fit to the device screen I had to decide whether I would sell Crystal or not. I didn’t ever think there would be a lack of users, but essentially it came down to the fact that I wanted Crystal in as many hands as possible. If I made it paid, I’m sure people would pay, but I knew I’d be much happier seeing more people use it, than less people giving me a moderately small amount of cash. Besides, I’m at univeristy and don’t need to make a living just yet. Instead I decided I’d release it free, generate some word of mouth and try to reach as many users as I could!

Not wanting to miss out on the potential gratitude of users though, I decided to add a donation section. It’s there if people want to use it, and if not, then fine by me. Since it was a page in Crystal though, I decided I’d at least make it look pretty. I asked a few of my friends to help me film fullscreen two videos. One video was just me - I thought any potential donater ought to see me - and the other was a big thank you, that donaters see after the payment. I’ve received some genuinely nice feedback from people for it :)

The Last (And Longest) Mile

Since release, Crystal has been covered on multiple news sites, appeared on Sketch’s mailing list and is used by small and large companies across the world. When you make an app that has an existing counterpart on another platform though (iOS in this case), there will always be the inevitable comparisons. Not long after release I started receiving requests for usb support and complaints that it doesn’t work on corporate networks (I mentioned earlier that corporate networks often block MDNS). I’d previously ruled out the possibility of usb support, but it’s not fair that iOS users get that privilege and not Android users too.

It took me a couple of days, but eventually I found a python sample on Github that used Android’s Accessory API to communicate over USB. The AOA (Android Open Accessory) protocol is a specification designed to allow hardware devices to communicate with Android devices. After successfully negotiating the protocol handshake, USB endpoints appear to the hardware device and input/output streams to the Android device to coummunicate over. Arn-o’s project simply showed how that handshake and communication could be implemented. In the spirit of modularisation, I refactored Crystal to completely separate the concern over what communication protocol the user chooses away from most of the code. After choosing to connect over USB, the internal server object is injected with a USB communication link rather than a websocket link, and all of the API requests operate as normal (with the exception of some extra api requests I added to support image transfers). I’d used Arn-o’s Python code for a while, but began porting it to Node JS because I knew I could package Node JS into a Mac app with the help of Electron. The Mac app acts as a super simple proxy. It talks with the Android app over the Android Accessory protocol, and to Sketch via its Websocket. All the data is just JSON, serialised into a bunch of bytes that are streamed over the connection. Any image requests the Mac app receives from the Android app, it requests locally from the Sketch webserver, before encoding as base64 and sending back across again.

After trumpeting the latest release of Crystal, I began to receive reports from some users over Twitter that USB wasn’t working on their device. It worked on mine and many other’s devices and I couldn’t quite nail the source of the issue. To may absolute joy, I found it wouldn’t run on Ruban’s phone, so I had a way of reliably reproducing the problem. After serious debugging I found the cause, but not the answer. I could write to the OutputStream, so send messages to the Mac app, but all messages coming from the Mac app, to Crystal were lost. My InputStream never received anything and I had absolutely no clue. No one else seemed to have the same problem using the USB Accessory API and I spent about two weeks trying all sorts of different approaches. I was ready to give up when I went away for the weekend. After having cleared my head, I came back to the problem and decided I’d find some more sample code from Android apps where Accessory mode definitively works. In the end it was a single line. Some Android devices handle InputStream.read(int bytesToRead) differently than others, and so the solution was to use a slightly different method that read as many bytes as it could, and returned the number it read. A very minor implementation detail, but one that cost me a full two weeks, and so I’m definitely mentioning here!

Some users couldn’t see their connected devices 😞

There was a secondary bug that caused the same issue but affected a different group of users. In the Android accessory documentation, it clearly states

The vendor ID should match Google’s ID (0x18D1)

Every USB device connected to a computer had a vendor ID and a product ID. The vendor ID indicates the manufacturer (Google uses 0x18D1, Apple uses 0x05ac), while the product ID is more specific and depends upon the manufacturer’s naming convention. So, following Google’s advice I was filtering out devices with an ID other than 0x18D1 because I didn’t want to try to connect to every mouse and keyboard people connect to their computer. It turns out that a very small subset of Android devices actually represent themselves with Google’s vendor ID, so the solution here was to find a large list of Android vendor IDs and allow them all.

Lastly, there was a gotcha that seemed to affect certain people trying to use the Mac app. People on corporate networks were still having issues connecting, and because of my 3-step connection process, I could tell it was the discovery of Sketch on their local machine. After a long train journey, the solution came to mind. I was still using bonjour to find the port of the local Sketch server from inside the Mac app because the port number changes randomly every time you restart Sketch (and rightly so). I could easily add some sort of field where users could enter the port number (because it’s visible from within Sketch), but that would completely destroy the “magic” of it all. During my testing this worked fine, but it had never failed because I couldn’t work out a way to disable Bonjour on my networks. When on a WiFi network, the Sketch server would appear under an IP address - I would check if it was the computer’s IP, if it was I had the port number. If the computer was completely disconnected from all networks then the Sketch server would appear on the loopback network interface, and I could get the port number the same way. The issue I’d overlooked, and that most users who needed USB support would experience was the following. In a network that doesn’t support bonjour, the computer that runs Sketch broadcasts to the network its address and port number. The network doesn’t support bonjour, so the broadcast doesn’t make it to any other computer, or, crucially, back to the computer that sent the broadcast. Even though the Mac app was running on the computer that sent the broadcast, it would only know of that if it received the computer’s own broadcast back, which it would on a normal network.

When mdns wasn’t enabled on the network, the Mac app couldn’t detect Sketch’s port number broadcast. The fix was to lookup which ports Sketch uses by querying the operating system.

The broadcast would not appear on the loopback network interface either, because the loopback interface wouldn’t be the computer’s primary interface (like it is when you disconnect/turn off wifi). I had to find another way to determine the Sketch port number. I ended up resorting to lsof. lsof enumerates all open file descriptors on the computer, and Mac being based on UNIX, uses file-based networking, so I could find open port references there. Simply calling lsof -a -c /sketchmir/i -i4 returns the ports on which Sketch is listening, and the solution to my problem.

Now, a few months on from the latest release everything is running smoothly, with crash-free rates hovering around 98%. I’ve finished what I set out to do and am over the moon that Crystal’s received the reception it has. I’ve learned a lot, so maybe my next project’s blog post won’t be quite so long!