What Happened to My Flash Drive?

I recently received a panicked call from a friend who couldn't find the flash drive he uses for multi-track field recording. Beyond me scolding him for only having one, it required a look under the hood to figure out what he might be able to get on short notice. The experience the two of us have shared has made us acutely aware that not all USB drives are created equal.

Taking that closer look at the requirements for his recording session, I also started to consider that we actually have different storage needs depending on the phase of our work. This post will concentrate on the needs of the capture phase.  Stay tuned for a discussion on the storage needs of the production and archiving stages.

Perhaps one of the first questions I should answer is why it might be hard to find an appropriate USB drive on short notice.  While it’s not all that hard to find a cheap USB flash drive or a portable hard drive (a 1TB WD my Passport is under $100 at several big box stores), there are a couple of best practices to consider.  First, as I said, they don’t all work for recording audio. Second, many portable audio recorders require a type of formatting (FAT32) that is difficult to do for capacities above 32GB. That means that 1,000 GB would leave a lot of wasted space (yes, it’s possible to format them with larger capacities, but as the size gets bigger there are some serious drawbacks.) Most importantly there’s a bit of “all your eggs in one basket” issue.  In the field, especially when powered, hard drives are rather fragile and it’s not hard to knock a portable one off a table. Even with less fragile solid state drives, there are plenty of things that can cause corruption or failure, so smaller capacities force us to move files off of the drives more often providing a layer of protection against catastrophic data loss.  

It is a fascinating twist of history that the professional audio world has essentially hitched a ride on the coattails of the photography market. Because of its relatively low data requirements compared to the massive streams required for video, we often think of audio as leading the way for new technology in media. In reality, it was the consumer hunger for "digital film" that funded the push to a portable replacement for bulky and fragile magnetic disks. By the late 90s, this technology had converged on a non-volatile version of memory called NAND. By the early 2000s, this NAND technology had mostly settled into two primary packages: the USB flash drive and the SD card.

In those early days, the internal electronics of these devices were virtually identical; a 64MB USB stick and a 64MB SD card were essentially using the same hardware. (Wow, when was 64MB considered plenty?) The markets for these devices, however, were already pulling them in different directions. USB flash drives were built for moving files between computers. SD cards were developed for storing photos on cameras.

As NAND technology evolved to support larger capacities, the underlying storage cells in both formats improved in density. While they have remained essentially the same in how they hold data, the surrounding technology diverged. Their internal architectures branched off to serve different markets, and the controllers managing that storage drifted apart. USB flash drives stayed focused on the "sneakernet" of the office world, where the engineering priority is providing high capacity at a low price for simple file transfers. In contrast, the camera market had to evolve to keep pace with the relentless data needs of high-definition video. This pressure forced the development of more robust controllers and the strict speed standards we see on SD cards today.

Several years ago, when I bought my RME UFX interface, it included a feature called DUREC. Released in 2010, it provided up to 60 channels of audio at 24-bit/48kHz via USB 2.0 or Firewire 400. The DUREC feature allowed for the recording of those 60 channels direct to a USB 2.0 drive.  There were a limited number of USB flash drives that could maintain the sustained write rate required (just under 10MB/sec). While drive capacities and the speed ratings have increased, even higher end flash drives today have trouble recording just a few channels of 24/48 audio without buffer problems.

To understand why a standard USB drive fails where an SD card succeeds, we have to look at how that "office-first" engineering handles data. Most USB flash drives are optimized for burst performance. The controller is designed to gulp down a chunk of data as fast as possible, using a small amount of high-speed cache as a "waiting room" before writing it to the slower storage cells. When it comes to a constant stream of audio data, eventually, that waiting room fills up. Because the controller wasn't engineered for sustained, heavy-lifting writes, it has to pause the incoming data stream to clear the cache. In a DAW or a field recorder, that half-second pause results in the dreaded Buffer Overrun error and an interrupted recording..

In contrast, SD cards are built for the video market, where a dropped frame is just as catastrophic as a dropped audio sample. Because of these demands, the controllers on these cards are engineered to manage data more efficiently over long periods. They ensure that the write speed never dips below a certain threshold. Even if the peak speed isn't as high as a "Pro" USB drive, the minimum is guaranteed.  In fact, SD cards can have several speed ratings, but that’s another show.

While thunderbolt and newer USB boast speeds of 40 Gbps, there is a lot of audio hardware that is perfectly happy running 32 channels of 24-bit audio (bidirectional) on the 480Mbs that USB 2.0 has to offer.  Come to think of it, there really aren’t a lot of multi-track hardware audio recorders on the market, but I suppose all of this suggests why, of the few that do exist, most of them record to SD card.

If you are one of the few recording multi-track audio direct to USB (Two RME UFX products are still current and provide the DUREC feature and the Soundcraft UI-24R can also record multi-channel files direct to USB), traditional USB flash drives are not really a viable solution. While there are plenty of traditional hard drives that use a USB interface and will handle the write requirements we need, it’s difficult to find capacities that are small enough to encourage good file management (and easily support FAT32 file systems).

Interestingly, you can build a highly reliable system by using a high-quality MicroSD card in a dedicated USB reader. By splitting these components, you get a controller specifically designed for the sustained demands of the SD standard. It’s important to keep in mind that shrinking from a regular SD to a MicroSD has its own physical trade-offs. MicroSD cards are smaller, so they have less surface area to dissipate heat. There’s a possibility for them to hit a thermal limit that forces the controller to slow down to protect the hardware. The risk that adding an additional mechanical interface introduces an additional point of failure (microSD to SD adapters or USB readers). I’ve been using this configuration for a few years now and have had more problems with traditional USB drives than this configuration, but it’s something to be aware of. For reference, I have been using a SanDisk MobileMate USB 3.0 microSD Card Reader that happens to be specified as UHS-1 (a speed rating specifically associated with SD cards). An added benefit is that the size is small enough that it runs a lower risk of getting bumped than a regular USB flash drive (although it is potentially easier to lose).

I hope this provides some insight into the use of storage options for the capture phase of media production.  While I’ve focused primarily on audio, the same holds true for video.  Look for my next post on clarifying the alphabet soup of data rates and SD card ratings.  Also coming soon is a post on the storage requirements we have during the production and archive phases.