digitisation expertise

Sharing insights and expertise around the digital transfer of audio and video assets

Measuring signals – challenges for the digitisation of sound and video

In a 2012 report entitled ‘Preserving Sound and Moving Pictures’ for the Digital Preservation Coalition’s Technology Watch Report series, Richard Wright outlines the unique challenges involved in digitising audio and audiovisual material. ‘Preserving the quality of the digitized signal’ across a range of migration processes that can negotiate ‘cycles of lossy encoding, decoding and reformatting is one major digital preservation challenge for audiovisual files’ (1).

Wright highlights a key issue: understanding how data changes as it is played back, or moved from location to location, is important for thinking about digitisation as a long term project. When data is encoded, decoded or reformatted it alters shape, therefore potentially leading to a compromise in quality. This is a technical way of describing how elements of a data object are added to, taken away or otherwise transformed when they are played back across a range of systems and software that are different from the original data object.

Time-Based-Corrector

To think about this in terms which will be familiar to people today, imagine converting an uncompressed WAV into an MP3 file. You then burn your MP3s onto a CD as a WAV file so it will play back on your friend’s CD player. The WAV file you started off with is not the same as the WAV file you end up with – its been squished and squashed, and in terms of data storage, is far smaller. While smaller file size may be a bonus, the loss of quality isn’t. But this is what happens when files are encoded, decoded and reformatted.

Subjecting data to multiple layers of encoding and decoding does not only apply to digital data. Take Betacam video for instance, a component analogue video format introduced by SONY in 1982. If your video was played back using composite output, the circuity within the Betacam video machine would have needed to encode it. The difference may have looked subtle, and you may not have even noticed any change, but the structure of the signal would be altered in a ‘lossy’ way and can not be recovered to it’s original form. The encoding of a component signal, which is split into two or more channels, to a composite signal, which essentially squashes the channels together, is comparable to the lossy compression applied to digital formats such as mp3 audio, mpeg2 video, etc.

UMatic-Time-Based-Corrector

A central part of the work we do at Greatbear is to understand the changes that may have occurred to the signal over time, and try to minimise further losses in the digitisation process. We use a range of specialist equipment so we can carefully measure the quality of the analogue signal, including external time based correctors and wave form monitors. We also make educated decisions about which machine to play back tapes in line with what we expect the original recording was made on.

If we take for granted that any kind of data file, whether analogue or digital, will have been altered in its lifetime in some way, either through changes to the signal, file structure or because of poor storage, an important question arises from an archival point of view. What do we do with the quality of the data customers send us to digitise? If the signal of a video tape is fuzzy, should we try to stabilise the image? If there is hiss and other forms of noise on tape, should we reduce it? Should we apply the same conservation values to audio and film as we do to historic buildings, such as ruins, or great works of art? Should we practice minimal intervention, use appropriate materials and methods that aim to be reversible, while ensuring that full documentation of all work undertaken is made, creating a trail of endless metadata as we go along?

Do we need to preserve the ways magnetic tape, optical media and digital files degrade and deteriorate over time, or are the rules different for media objects that store information which is not necessarily exclusive to them (the same recording can be played back on a vinyl record, a cassette tape, a CD player, an 8 track cartridge or a MP3 file, for example)? Or should we ensure that we can hear and see clearly, and risk altering the original recording so we can watch a digitised VHS on a flat screen HD television, in line with our current expectations of media quality?

Time-Based-Correctors

Richard Wright suggests it is the data, rather than operating facility, which is the important thing about the digital preservation of audio and audiovisual media.

‘These patterns (for film) and signals (for video and audio) are more like data than like artefacts. The preservation requirement is not to keep the original recording media, but to keep the data, the information, recovered from that media’ (3).

Yet it is not always easy to understand what parts of the data should be discarded, and which parts should kept. Audiovisual and audio data are a production of both form and content, and it is worth taking care over the practices we use to preserve our collections in case we overlook the significance of this point and lose something valuable – culturally, historically and technologically.

Posted by debra in audio tape, digitisation expertise, video tape, 0 comments

Copying U-matic tape: digitise via dub connector or composite video?

umatic dub to y/c converter detail

Digitising legacy and obsolete video formats in essence is simple but the technical details make the process more complex. Experience and knowledge are therefore needed to make the most appropriate choices for the medium.

The U-matic video format usually had two types of video output, composite and a y/c type connector that Sony named ‘Dub’. Originally designed as a higher quality method to make analogue ‘dubs’, or for connections in an edit suite, the Dub connector offers a higher performance signal path for the video signal.

It would make sense to use the higher quality dub output when digitising U-matic tapes but here lies the problem. Firstly the connector uses the larger 7 pin y/c type connector that can be quite hard to find connectors for.

Secondly and most significantly, the chrominance subcarrier frequency is not the standard PAL  4.43Mhz but down converted by U-matic recorders to 0.686Mhz for low band recordings and 0.984Mhz for high band recordings.

What this means in practice is that you’ll only get a monochrome image using the U-matic dub connector unless you can find a way to convert the chroma subcarrier frequency back to 4.43Mhz.

There are several solutions:

  1. Convert this Dub signal chroma frequency using one of a few older Timebase Correctors / Frame Synchronisers  from the U-matic era.
    These are now rare and often have other other faults that would degrade the signal.
  2. Take the Luma and Chroma signals at the correct frequency directly from certain test points on the circuit boards inside the machines.
    This can work well but is a slightly ‘messy’ solution and makes it hard to swap machines around, which is a necessity with older hardware.
  3. Convert the dub signal using a dedicated external dub – y/c converter circuit.
    This is our preferred solution that works well technically. It is flexible enough to swap around to different machines easily. It is also a relatively simple circuit that is easy to repair and doesn’t subject the video signal to  unnecessary extra processing.

Below are two stills taken from a Apple ProRes recording from a Low Band PAL U-matic tape.
The first image is via the Dub connecter but converted to PAL Y/C.
The second images is via the Composite video out.

kieran prendiville umatic screenshot dub connector

 

kieran prendiville umatic screenshot composite video connector

It’s clear from the images that there is more fine detail in the picture from the U-matic Dub version. The pattern / texture in the jacket and the texture and tone in the face is more detailed.  In contrast, the version digitised through the Composite video connector has less noise but due to the extra encoding and decoding there is less detail and more ‘blurring’.

While less noise may be preferable in some instances, having the option to choose between these two is always better. It’s this kind of attention detail and investment in equipment and knowledge that we are proud of and makes us a preferred supplier of digitising services for U-matic video tape.

Posted by greatbear in digitisation expertise, video tape, 7 comments

Delivery formats – to compress or not compress

Screenshot of software encoding a file to MP3 used at the Great Bear

After we have migrated your analogue or digital tape to a digital file, we offer a range of delivery formats.

For video, using the International Association of Sound & Audiovisual Archives Guidelines for the Preservation of Video Recordings, as our guide, we deliver FFV1 lossless files or 10-bit uncompressed video files in .mkv or QuickTime compatible .mov containers. We add viewing files as H264 encoded .mp4 files or DVD. We’ll also produce any other digital video files, according to your needs, such as AVI in any codec; any MacOS, Windows or GNU/Linux filesystem (HFS+, NTFS or EXT3.

For audio we offer Broadcast WAV (B-WAV) files on hard drive or optical media (CD) at 16 bit/44.1 kHz (commonly used for CDs) or 24 bit/96 kHz (which is the minimum recommended archival standard) and anything up to 24 bit / 192 kHz. We can also deliver access copies on CD or MP3 (that you could upload to the internet, or listen to on an ipod, for example).

Why are there so many digital file types and what distinguishes them from each other?

The main difference that is important to grasp is between an uncompressed digital file and a compressed one.

On the JISC Digital Media website, they describe uncompressed audio files as follows:

‘Uncompressed audio files are the most accurate digital representation of a soundwave, but can also be the   most resource-intensive method of recording and storing digital audio, both in terms of storage and management. Their accuracy makes them suitable for archiving and delivering audio at high resolution, and working with audio at a professional level, and they are the “master” audio format of choice.’

Why uncompressed?

As a Greatbear client you may wonder why you need a large, uncompressed digital file if you only want to listen to your old analogue and digital tapes again. The simple answer is: we live in an age where information is dynamic rather static. An uncompressed digital recording captured at a high bit and kHz rate is the most stable media format you can store your data on. Technology is always changing and evolving, and not all types of digital files that are common today are safe from obsolescence.

It is important to consider questions of accessibility not only for the present moment, but also for the future. There may come a time when your digitised audio or video file needs to be migrated again, so that it can be played back on whatever device has become ‘the latest thing’ in a market driven by perpetual innovation. It is essential that you have access to the best quality digital file possible, should you need to transport your data in ten, fifteen or twenty years from now.

Compression and compromise?

Uncompressed digital files are sound and vision captured in their purest, ‘most accurate’ form. Parts of the original recording are not lost when the file is converted or saved. When a digital file is saved to a compressed, lossy format, some of its information is lost. Lossy compression eliminates ‘unnecessary’ bits of information, tailoring the file so that it is smaller. You can’t get the original file back after it has been compressed so you can’t use this sort of compression for anything that needs to be reproduced exactly. However it is possible to compress files to a lossless format, which does enable you to recreate the original file exactly.

In our day to day lives however we encounter far more compressed digital information than uncompressed.

There would be no HD TV, no satellite TV channels and no ipods/ MP3 players without compressed digital files. The main point of compression is to make these services affordable. It would be incredibly expensive, and it would take up so much data space, if the digital files that were streamed to televisions were uncompressed.

While compression is great for portability, it can result in a compromise on quality. As Simon Reynolds writes in his book Retromania: Pop Culture’s Addiction to its Own Past about MP3 files:

‘Every so often I’ll get the proper CD version of an album I’ve fallen in love with as a download, and I’ll get a rude shock when confronted by the sense of dimension and spatiality in the music’s layers, the sculpted force of the drums, the sheer vividness of the sound. The difference between CD and MP3 is similar to that between “not from concentrate” orange juice and juice that’s been reconstituted from concentrate. (In this analogy vinyl would be ‘freshly squeezed, perhaps). Converting music to MP3 is a bit like the concentration process, and its done for much the same reason: it’s much cheaper to transport concentrate because without the water it takes up a lot loss volume and it weighs a lot less. But we can all taste the difference.’

As a society we are slowly coming to terms with the double challenge of hyper consumption and conservation thrown up by the mainstreaming of digital technology. Part of that challenge is to understand what happens to the digital data we use when we click ‘save as,’ or knowing what decisions need to be made about data we want to keep because it is important to us as individuals, or to wider society.

At Greatbear we can deliver digital files in compressed and uncompressed formats, and are happy to offer a free consultation should you need it to decide what to do with your tape based digital and analogue media.

Posted by debra in audio tape, digitisation expertise, video tape, 0 comments

Convert, Join, re encode AVCHD .MTS files in Ubuntu Linux

convert, encode and join avchd files in linux

One of our audio and video archive customers has a large collection of AVCHD video files that are stored in 1.9GB ‘chunks’ as xxxxx.MTS files. All these files are of 60 minute and longer duration and must be joined, deinterlaced, re encoded to a suitable size and bitrate then uploaded for online access.

This is quite a task in computer time and file handling. These small domestic cameras produce good HD movies for a low cost but the compression to achieve this is very high and does not give you a file that is easily edited. The .MTS files are MPEG transport stream containers for H264 encoded video.

There are some proprietary solutions for MacOS X and Windows that will repackage the .MTS files into .MOV Quicktime containers that can be accessed by MacOS X or re-encoded to a less compressed format for editing with Final Cut Pro or Premiere. We didn’t need this though, just a reliable and  quick open source workflow.

  1. The first and most important issue is to rejoin the camera split files.
    These cameras use FAT32 file systems which cannot handle individual files larger than 2GB so they split the .MTS video file into chunks. As each chunk in a continuous sequence references the other chunks these must be joined in the correct order. This is easily achieved with the cat command.
  2. The rejoined .MTS files can now be reencoded to a more manageable size using open source software such as Handbrake. We also needed to deinterlace our footage as it was shot interlaced and it would be accessed on progressive displays. This will increase the encoding time but without it any movement will look odd with visible artifacts.
  3. Finding the ‘sweet spot’ for encoding can be time consuming but in this case was important as projected text needed to be legible but the file sizes kept manageable for reasonable upload times!

 

Posted by greatbear in digitisation expertise, video tape, 0 comments

Audio data recovery from external USB drive using ddrescue

High resolution audio and video digital tape conversions can use large amounts of computer storage. 8 bit uncompressed Standard Definition (SD) PAL video runs at 70 GB per hour and 24 bit 96 kHz audio files at 2 GB per hour.

As a result of this many of our analogue to digital tape transfers require the use of external storage, usually USB 2.0 portable hard drives, to supply the copied digital transfers back to the customer. Some drives supplied by customers have not been of great quality and not designed to be sent about in the post. One such drive we had recently, a Sony Vaio branded 2.5″ USB drive wouldn’t copy certain directories of important files with the Mac OS Finder or the Windows Explorer. While most of the drive copied this certain folder always resulted in a crashed computer!

Thanks to GNU/Linux we have a bit more power and information at our disposal about hard drives and IDE or USB interfaces. It’s always best practice to copy as much information from the drive or mirror it before attempting any other types of data recovery or file system repair. Using the standard dd

Posted by greatbear in digitisation expertise, 0 comments