← All posts

// Blog

Audio and Video formats for the Clarify API

Clarify

Originally published on the Clarify.io blog. View archived copy.

Sound wave
Audio and video comes in all shapes and sizes. More importantly, it comes in different codecs (or encodings) which each capture different types and amounts of information. When we look at processes like Automatic Speech Recognition (ASR) or Natural Language Processing (NLP), the more information we have, the better. It allows us to figure out the gaps and estimate what might have been said.

In short, we support a ton of audio and video formats. Here’s what we currently handle: aac, flac, m4v, mpeg, mp3, mp4, mov, ogg, wav, and webm.

Unfortunately, not every audio or video file comes with enough data to take advantage of these processes but we do have some guidelines and suggestions to help you get started:

  • First, your audio and video should be the highest quality possible. If you have the raw uncompressed audio or video, that will give better results than a lossy format like mp3.
  • On the other side, remember that the required processing time is directly related to the quality of the file. Higher quality means more data means more processing. As a general rule, stereo quality audio – 16 or 32 bit audio sampled at 44.1 kHz – is going to be sufficient. This is the same quality you might get off a CD.. if you’re familiar with one of those.
  • Next, don’t try to re-sample the audio for a particular ASR. Each one has its own configuration, sampling information, and nuances. Instead our system will determine the best approach by analyzing your audio – both acoustic models and languages – and re-sample as needed, if needed.
  • Finally, don’t try to “upsample” or modify the audio or video. Most likely you’ll introduce artifacts and other oddities which end up making processing harder. More importantly, you’re attempting to “fill in the gaps” for the higher sampling rate and we do the same. It’s like introducing an Off by One Error in your code.. twice.

The key thing to remember is that the ASR and NLP are only as good as the data they start with. It’s very much a “garbage in, garbage out” scenario. The better information you can get to our systems, the more likely you are to get great results.

And as always, if you have any questions, don’t hesitate to drop us a note: support@clarify.io or @ClarifyAPI.