access

Digitising small audiovisual collections: making decisions and taking action

Deciding when to digitise your magnetic tape collections can be daunting.

The Presto Centre, an advocacy organisation working to help ‘keep audiovisual content alive,’ have a graphic on their website which asks: ‘how digital are our members?’

They chart the different stages of ‘uncertainty,’ ‘awakening’, ‘enlightenment’, ‘wisdom’ and ‘certainty’ that organisations move through as they appraise their collections and decide when to re-format to digital files.

Similarly, the folks at AV Preserve offer their opinion on the ‘Cost of Inaction‘ (COI), arguing that ‘incorporating the COI model and analyses into the decision making process around digitization of legacy physical audiovisual media helps organizations understand the implications and make well-informed decisions.’

They have even developed a COI calculator tool that organisations can use to analyse their collections. Their message is clear: ‘the cost of digitization may be great, but the cost of inaction may be greater.’

Digitising small-medium audiovisual collections

For small to medium size archives, digitising collections may provoke worries about a lack of specialist support or technical infrastructure. It may be felt that resources could be better used elsewhere in the organisation. Yet as we, and many other people working with audiovisual archives often stress, the decision to transfer material stored on magnetic tape has to be made sooner or later. With smaller archives, where funding is limited, the question of ‘later’ is not really a practical option.

Furthermore, the financial cost of re-formatting audiovisual archives is likely to increase significantly in the next five-ten years; machine obsolescence will become an aggravated problem and it is likely to take longer to restore tapes prior to transfer if the condition of carriers has dramatically deteriorated. The question has to be asked: can you afford not to take action now?

If this describes your situation, you might want to hear about other small to medium sized archives facing similar problems. We asked one of our customers who recently sent in a comparatively small collection of magnetic tapes to share their experience of deciding to take the digital plunge.

We are extremely grateful for Annaig from the Medical Mission Sisters for answering the questions below. We hope that it will be useful for other archives with similar issues.

threadimg-eiaj-half-inch-video-tape 1. First off, please tell us a little bit about the Medical Mission Sisters Archive, what kind of materials are in the collection?

The Medical Mission Sisters General Archives include the central archives of the congregation. They gather all the documents relating to the foundation and history of the congregation and also documents relating to the life of the foundress, Anna Dengel. The documents are mainly paper but there is a good collection of photographs, slides, films and audio documents. Some born digital documents are starting to enter the archives but they are still few.

2. As an archive with a modest collection of magnetic tapes, why did you decide to get the materials digitised now? Was it a question of resources, preservation concerns, access request (or a mixture of all these things!)

The main reason was accessibility. The documents on video tapes or audio tapes were the only usable ones because we still had machines to read them but all the older ones, or those with specific formats,  where lost to the archives as there was no way to read them and know what was really on the tapes. Plus the Medical Mission Sisters is a congregation where Sisters are spread out on 5 continents and most of the time readers don’t come to the archives but send me queries by emails where I have to respond with scanned documents or digital files. Plus it was obvious that some of the tapes were degrading as that we’d better have the digitisation sooner than later if we wanted to still be able to read what was on them. Space and preservation was another issue. With a small collection but varied in formats, I had no resources to properly preserve every tape and some of the older formats had huge boxes and were consuming a lot of space on the shelves. Now, we have a reasonably sized collection of CDs and DVDs, which is easy to store in good conditions and is accessible everywhere as we can read them on computer here and I can send them to readers via email.

3. Digital preservation is a notoriously complex, and rapidly evolving field. As a small archive, how do you plan to manage your digital assets in the long term? What kinds of support, services and systems are your drawing on to design a system which is robust and resilient?

At the moment the digital collection is so small that it cannot justify any support service or system. So I have to build up my own home made system. I am using the archives management software (CALM) to enter data relating to the conservation of the CDs or DVDs, dates of creation, dates to check them and I plan to have regular checks on them and migrations or copies made when it will prove necessary.

4. Aside from the preservation issue, what are your plans to use the digitised material that Greatbear recently transferred?

It all depends on the content of the tapes. But I’ve already spotted a few documents of interest, and I haven’t been through everything yet. My main concern now is to make the documents known and used for their content. I was already able to deliver a file to one of the Sisters who was working on a person related to the foundation of the congregation, the most important document on her was an audio file that I had just received from Greatbear, I was able to send it to her. The document would have been unusable a few weeks before. I’ve come across small treasures, like a film, probably made by the foundress herself, which nobody was aware of. The Sisters are celebrating this year the 90th anniversary of their foundation. I plan to use as many audio or video documents as I can to support the events the archives are going to be involved into.

***

What is illuminating about Annaig’s answers is that her archive has no high tech plan in place to manage the collection – her solutions for managing the material very much draw on non-digital information management practices.

The main issues driving the decision to migrate the materials are fairly common to all archives: limited storage space and accessibility for the user-community.

What lesson can be learnt from this? Largely, that if you are trained as an archivist, you are likely to already have the skills you need to manage your digital collection.

So don’t let the more bewildering aspects of digital preservation put you off. But do take note of the changing conditions for playing back and accessing material stored on magnetic tape. There will come a time when it will be too costly to preserve recordings on a wide variety of formats – many of such formats we can help you with today.

If you want to discuss how Greatbear can help you re-format your audiovisual collections, get in touch and we can explore the options.

If you are a small-medium size archive and want to share your experiences of deciding to digitise, please do so in the comment box below.

Posted by debra in audio / video heritage, audio tape, video tape, 0 comments

Open Source Solutions for Digital Preservation

In a technological world that is rapidly changing how can digital information remain accessible?

One answer to this question lies in the use of open source technologies. As a digital preservation strategy it makes little sense to use codecs owned by Mac or Windows to save data in the long term. Propriety software essentially operate like closed systems and risk compromising access to data in years to come.

Linux Operating System

It is vital, therefore, that the digitisation work we do at Great Bear is done within the wider context of digital preservation. This means making informed decisions about the hardware and software we use to migrate your tape-based media into digital formats. We use a mixture of propriety and open source software, simply because it makes our a bit life easier. Customers also ask us to deliver their files in propriety formats. For example, Apple pro res is a really popular codec that doesn’t take up a lot of data space so our customers often request this, and of course we are happy to provide it.

Using open systems definitely has benefits. The flexibility of Linux, for example, enables us to customise our digitisation system according to what we need to do. As with the rest of our work, we are keen to find ways to keep using old technologies if they work well, rather than simply throwing things away when shiny new devices come on the market. There is the misconception that to ingest vast amounts of audio data you need the latest hardware. All you need in fact is a big hard drive, flexible, yet reliable, software and an operating system that doesn’t crash so it can be left to ingest for 8 hours or more. Simple! Examples of open source software we use is the sound processing programme SoX. This saves us a lot of time because we are able to write scripts for the programme that can be used to batch process audio data according to project specifications.

Openness in the digital preservation world

Within the wider digital preservation world open source technologies are also used widely. From digital preservation tools developed by projects such as SCAPE and the Open Planets Foundation, there are plenty of software resources available for individuals and organisations who need to manage their digital assets. It would be naïve, however, to assume that the practice of openness here, and in other realms of the information economy, are born from the same techno-utopian impulse that propelled the open software movement from the 1970s onwards. The SCAPE website makes it clear that the development of open source information preservation tools are ‘the best approach given the substantial public investment made at the European and national levels, and because it is the most effective way to encourage commercial growth.’

What would make projects like SCAPE and Open Planets even better is if they thought about ways to engage non-specialist users who may be curious about digital preservation tools but have little experience of navigating complex software. The tools may well be open, but the knowledge of how to use them are not.

Openness, as a means of widening access to technical skills and knowledge, is the impulse behind the AV Artifact Atlas (AVAA), an initiative developed in conjunction with the community media archive project Bay Area Video Coalition. In a recent interview on the Library of Congress’ Digital Preservation Blog, Hannah Frost, Digital Library Services Manager at Stanford Libraries and Manager, Stanford Media Preservation Lab explains the idea behind the AVAA.

‘The problem is most archivists, curators and conservators involved in media reformatting are ill-equipped to detect artifacts, or further still to understand their cause and ensure a high quality job. They typically don’t have deep training or practical experience working with legacy media. After all, why should we? This knowledge is by and large the expertise of video and audio engineers and is increasingly rare as the analogue generation ages, retires and passes on. Over the years, engineers sometimes have used different words or imprecise language to describe the same thing, making the technical terminology even more intimidating or inaccessible to the uninitiated. We need a way capture and codify this information into something broadly useful. Preserving archival audiovisual media is a major challenge facing libraries, archives and museums today and it will challenge us for some time. We need all the legs up we can get.’

The promise of openness can be a fraught terrain. In some respects we are caught between a hyper-networked reality, where ideas, information and tools are shared openly at a lightning pace. There is the expectation that we can have whatever we want, when we want it, which is usually now. On the other side of openness are questions of ownership and regulation – who controls information, and to what ends?

Perhaps the emphasis placed on the value of information within this context will ultimately benefit digital archives, because there will be significant investment, as there already has been, in the development of open resources that will help to take care of digital information in the long term.

Posted by debra in audio tape, digitisation expertise, video tape, 0 comments

Digital Preservation – Establishing Standards and Challenges for 2014

2014 will no doubt present a year of new challenges for those involved in digital preservation. A key issue remains the sustainability of digitisation practices within a world yet to establish firm standards and guidelines. Creating lasting procedures capable of working across varied and international institutions would bring some much needed stability to a profession often characterized by permanent change and innovation.

In 1969 The EIAJ-1 video tape was developed by the Electronic Industries Association of Japan. It was the first standardized format for industrial/non-broadcast video tape recording. Once implemented it enabled video tapes to be played on machines made by different manufacturers and it helped to make video use cheaper and more widespread, particularly within a domestic context.

Close up of tape machine on the 'play', 'stop', 'rewind' button

The introduction of standards in the digitisation world would of course have very little impact on the widespread use of digital technologies which are, in the west, largely ubiquitous. It would however make the business of digital preservation economically more efficient, simply because organisations would not be constantly adapting to change. For example, think of the costs involved in keeping up with rapid waves of technological transformation: updating equipment, migrating data and ensuring file integrity and operability are maintained are a few costly and time consuming examples of what this would entail.

Although increasingly sophisticated digital forensic technology can help to manage some of these processes, highly trained (real life!) people will still be needed to oversee any large-scale preservation project. Within such a context resource allocation will always have to account for these processes of adaptation. It has to be asked then: could this money, time and energy be practically harnessed in other, more efficient ways? The costs of non-standardisation becomes ever more pressing when we consider the amount of the digital data preserved by large institutions such as the British Library, whose digital collection is estimated to amass up to 5 petabytes (5000 terabytes) by 2020. This is not a simple case of updating your iphone to the next model, but an extremely complex and risky venture where the stakes are high. Do we really want to jeopardise rich forms cultural heritage in the name of technological progress?

The US-based National Digital Stewardship Alliance (NDSA) National Agenda for Digital Stewardship 2014 echoes such a sentiment. They argue that ‘the need for integration, interoperability, portability, and related standards and protocols stands out as a theme across all of these areas of infrastructure development’ (3). The executive summary also stresses the negative impact rapid technological change can create, and the need to ‘coordinate to develop comprehensive coverage on critical standards bodies, and promote systematic community monitoring of technology changes relevant to digital preservation.’ (2)

File Format Action Plans

One step on the way to more secure standards is the establishment of File Format Action Plans, a practice which is being increasingly recommended by US institutions. The idea behind developing a file format action plan is to create a directory of file types that are in regular use by people in their day to day lives and by institutions. Getting it all down on paper can help us track what may be described as the implicit user-standards of digital culture. This is the basic idea behind Parsimonious Preservation, discussed on the blog last year: that through observing trends in file use we may come to the conclusion that the best preservation policy is to leave data well alone since in practice files don’t seem to change that much, rather than risk the integrity of information via constant intervention.

As Lee Nilsson, who is currently working as a National Digital Stewardship Resident at the US Library of Congress writes, ‘specific file format action plans are not very common’, and when created are often subject to constant revision. Nevertheless he argues that devising action plans can ‘be more than just an “analysis of risk.” It could contain actionable information about software and formats which could be a major resource for the busy data manager.’

Other Preservation Challenges

Analogue to Digital Converter close up What are the other main challenges facing ‘digital stewards’ in 2014? In a world of exponential information growth, making decisions about what we keep and what we don’t becomes ever more pressing. When whole collections cannot be preserved digital curators are increasingly called upon to select material deemed representative and relevant. How is it possible to know now what material needs to be preserve for posterity? What values inform our decision making?

To take an example from our work at Great Bear: we often receive tapes from artists who have achieved little or no commercial success in their life times, but whose work is often of great quality and can tell us volumes about a particular community or musical style. How does such work stand up against commercially successful recordings? Which one is more valuable? The music that millions of people bought and enjoyed or the music that no one has ever heard?

Ultimately these questions will come to occupy a central concern for digital stewards of audio data, particularly with the explosion of born-digital music cultures which have enabled communities of informal and often non-commercial music makers to proliferate. How is it possible to know in advance what material will be valuable for people 20, 50 or 100 years from now? These are very difficult, if not impossible questions for large institutions to grapple with, and take responsibility for. Which is why, as members of a digital information management society, it is necessary to empower ourselves with relevant information so we can make considered decisions about our own personal archives.

A final point to stress is that among the ‘areas of concern’ for digital preservation cited by the NDSA, moving image and recorded sound figure highly, alongside other born-digital content such as electronic records, web and social media. Magnetic tape collections remain high risk and it is highly recommended that you migrate this content to a digital format as soon as possible. While digitisation certainly creates many problems as detailed above, magnetic tape is also threatened by physical deterioration and its own obsolescence challenges, in particular finding working machines to play back tape on. The simple truth is, if you want to access material in your tape collections it needs now to be stored in a resilient digital format. We can help, and offer other advice relating to digital information management, so don’t hesitate to get in touch.

Posted by debra in audio tape, video tape, 0 comments

Digital Preservation and Copyright

Most customers who send us tape to digitise own the copyright of their recording: it is material they have created themselves, be it music, spoken word or film.

Occasionally customers are not so sure if they own the full copyright to their recordings. This is because a single piece of work can have multiple copyright holders.

For example, films and songs can have many different contributors, such as the person who made the recording, the songwriter and performers. There are performing rights royalties which are paid to a songwriter, composer or publisher whenever their music is played or performed in any public space or place; mechanical rights royalties which are paid to the songwriter, composer or publisher when music is reproduced as a physical product or for broadcast or online, and performers rights royalties which are paid to the people performing on the record. It can seem like a bit of a minefield, and you do have to be really careful, particularly if want to re-publish the works in a commercial context.

A box of full of reel to reel tapes, the spools are dirty because of how they were stored

A collection of tapes that include original recordings made by the customer

The simple truth is, if you do not have full permission of all copyright holders, you would break the law if you digitised a tape and re-published it commercially.

Copyright, Intellectual Property and Digital Preservation is a tricky area to negotiate. Currently ‘there is still no exception in UK law for preservation copying. For materials which are still in copyright, permissions should be sought from copyright holders prior to any copying being done. This area is under consideration though with museums, libraries and archives lobbying for change’ (Jisc Digital Media).

What this means basically is that archives, libraries and museums are effectively restricted in how much material they can legally preserve in digital form. Andrew Charlesworth explains in a very useful report for the Digital Preservation Coalition on ‘Intellectual Property Rights for Digital Preservation’ (2012)

‘In “Chapter III: Acts permitted in relation to copyright works”, the Copyright Designs and Patents Act 1988 provides for a series of permissible activities that would otherwise be barred for breach of a rights holder’s exclusive rights. These include the “fair dealing provisions” which, for example, state that making transient copies is an integral and essential part of certain technological processes (s.28), and using all or part of a copyright work for non-commercial research or private study (s.29), criticism or review, or reporting current events (s.30), do not constitute infringements’ (11).

Clearly copyright law as it stands places immense restrictions in a digital environment where copying and sharing all kinds of things is pretty much the norm. What are the arguments then for changing copyright laws? In Imagine there is no copyright and cultural conglomerates too by Joost Smiers and Marieke Van Schinjdel, published by the Institute of Network CulturesTheory on Demand series, they argue that removing copyright from cultural products will ensure that ‘our past and present heritage of cultural expression, our public domain of artistic creativity and knowledge will no longer be privatised’ (6).

Making cultural heritage publicly available is an argument for transforming current copyright laws across the range of political positions. While Smiers and Van Schinjdel interpret privatisation embedded in copyright law as linked to commercial power, the implicit argument in the DPC report is that opening up current restrictions can only be good for business. In this particular domain we see how the value of archival information has shifted in the digital landscape, so that it is increasingly seen as a resource through which money can be made.

Copyright symbol attached to a padlock A transformation of copyright laws would not necessarily lead to a weakening of commercial interests as Smiers and Van Schinjdel speculate, but would most probably enable the re-use of information across a range of profit and profit-making initiatives. Charlesworth insists we are ‘clinging to copyright practices that reflect outdated business models rather than attempting to establish new practices to address the prevailing mixed analogue/digital environment’ (7).

The digital information revolution has required all sectors of society to change how they relate to, use, record, save and consume information. While we have all become, to a lesser or more degree, record keepers, this brief survey of copyright law may help us appreciate the challenges professional archivists face in negotiating this complex area. After all, ‘life would be much simpler for archivists if the law relating to the preservation of copyright works in general, and digital works in particular, was both clarified and, where necessary, extended to permit more robust strategies for collection, preservation and reuse of copyright works’ (5).

 

Posted by debra in audio tape, video tape, 0 comments

Parsimonious Preservation – (another) different approach to digital information management

We have been featuring various theories about digital information management on this blog in order to highlight some of the debates involved in this complex and evolving field.

To offer a different perspective to those that we have focused on so far, take a moment to consider the principles of Parsimonious Preservation that has been developed by the National Archives, and in particular advocated by Tim Gollins who is Head of Preservation at the Institution.

racks of servers storing digital information

In some senses the National Archives seem to be      bucking the trend of panic, hysteria and (sometimes)  confusion that can be found in other literature relating  to digital information management. The advice given in  the report, ‘Putting Parsimonious Preservation into  Practice‘, is very much advocating a hands-off, rather  than hands-on approach, which many other  institutions, including the British Library, recommend.

The principle that digital information requires  continual interference and management during its life  cycle is rejected wholesale by the principles of  parsimonious preservation, which instead argues that  minimal intervention is preferable because this entails  ‘minimal alteration, which brings the benefits of  maximum integrity and authenticity’ of the digital data object.

As detailed in our previous posts, cycles of coding and encoding pose a very real threat to digital data. This is because it can change the structure of the files, and risk in the long run compromising the quality of the data object.

Minimal intervention in practice seems here like a good idea – if you leave something alone in a safe place, rather than continually move it from pillar to post, it is less likely to suffer from everyday wear and tear. With digital data however, the problem of obsolescence is the main factor that prevents a hands-off approach. This too is downplayed by the National Archives report, which suggests that obsolescence is something that, although undeniably a threat to digital information, it is not as a big a worry as it is often presented.

Gollins uses over ten years of experience at the National Archives, as well as the research conducted by David Rosenthal, to offer a different approach to obsolescence that takes note of the ‘common formats’ that have been used worldwide (such as PDF, .xls and .doc). The report therefore concludes ‘that without any action from even a national institution the data in these formats will be accessible for another 10 years at least.’

10 years may seem like a short period of time, but this is the timescale cited as practical and realistic for the management of digital data. Gollins writes:

‘While the overall aim may be (or in our case must be) for ―permanent preservation […] the best we can do in our (or any) generation is to take a stewardship role. This role focuses on ensuring the survival of material for the next generation – in the digital context the next generation of systems. We should also remember that in the digital context the next generation may only be 5 to10 years away!’

It is worth mentioning here that the Parsimonious Preservation report only includes references to file extensions that relate to image files, rather than sound or moving images, so it would be a mistake to assume that the principle of minimal intervention can be equally applied to these kinds of digital data objects. Furthermore, .doc files used in Microsoft Office are not always consistent over time – have you ever tried to open a word file from 1998 on an Office package from 2008? You might have a few problems….this is not to say that Gollins doesn’t know his stuff, he clearly must do to be Head of Preservation at the National Archives! It is just this ‘hands-off, don’t worry about it’ approach seems odd in relation to the other literature about digital information management available from reputable sources like The British Library and the Digital Preservation Coalition. Perhaps there is a middle ground to be struck between active intervention and leaving things alone, but it isn’t suggested here!

For Gollins, ‘the failure to capture digital material is the biggest single risk to its preservation,’ far greater than obsolescence. He goes on to state that ‘this is so much a matter of common sense that it can be overlooked; we can only preserve and process what is captured!’ Another issue here is the quality of the capture – it is far easier to preserve good quality files if they are captured at appropriate bit rates and resolution. In other words, there is no point making low resolution copies because they are less likely to survive the rapid successions of digital generations. As Gollins writes in a different article exploring the same theme, ‘some will argue that there is little point in preservation without access; I would argue that there is little point in access without preservation.’

Diagram explaining how emulation works to make obsolete computers available on new machines

This has been bit of a whirlwind tour through a very interesting and thought provoking report that explains how a large memory institution has put into practice a very different kind of digital preservation strategy. As Gollins concludes:

‘In all of the above discussion readers familiar with digital preservation literature will perhaps be surprised not to see any mention or discussion of “Migration” vs. “Emulation” or indeed of ―“Significant Properties”. This is perhaps one of the greatest benefits we have derived from adopting our parsimonious approach – no such capability is needed! We do not expect that any data we have or will receive in the foreseeable future (5 to 10 years) will require either action during the life of the system we are building.’

Whether or not such an approach is naïve, neglectful or very wise, only time will tell.

Posted by debra in audio tape, 2 comments

Digital Preservation – Planning for the Long Term

There are plenty of reflections on the Great Bear tape blog about the fragility of digital data, and the need to think about digitisation as part of a wider process of data migration your information will need to make in its lifetime.

We have also explored how fast moving technological change can sometimes compromise our capacity to construct long term strategies for the survival of digital data.

digital-data-stream-visualisation (2)

This why it is so important that organisations such as the Digital Preservation Coalition, founded in February 2002, articulate a vision that aims to make ‘digital memory accessible tomorrow.’ Their website goes on to say:

Our generation has invested as never before in digital resources and we’ve done so because of the opportunity they bring. They have grown in volume, complexity and importance to the point that our children are baffled by the inefficiencies of the analogue age. Pervasive, fluid and fragile: digital data is a defining feature of our age. Industry, commerce, government, law, research, health, social care, education, the creative industries, the heritage sector and private life depend on digital materials to satisfy ubiquitous information needs and expectations. Digital preservation is an issue which all organisations, particularly in the knowledge sector, will need to address sooner or later.

As providers of a digitisation service it is important for us to understand digitisation in line with the ideas articulated above. This means creating high quality, uncompressed files that will make it as easy as possible for data migrations to happen in the future should they need to.

Organisations such as the Digital Preservation Coalition are providing sensible advice and creating forums for learning and debate about the problems and possibilities of digital preservation.

These are two things that are needed as we come to navigate an information environment heavily populated by ‘pervasive, fluid and fragile’ digital data.

 

Posted by debra in audio tape, video tape, 1 comment

Convert, Join, re encode AVCHD .MTS files in Ubuntu Linux

convert, encode and join avchd files in linux

One of our audio and video archive customers has a large collection of AVCHD video files that are stored in 1.9GB ‘chunks’ as xxxxx.MTS files. All these files are of 60 minute and longer duration and must be joined, deinterlaced, re encoded to a suitable size and bitrate then uploaded for online access.

This is quite a task in computer time and file handling. These small domestic cameras produce good HD movies for a low cost but the compression to achieve this is very high and does not give you a file that is easily edited. The .MTS files are MPEG transport stream containers for H264 encoded video.

There are some proprietary solutions for MacOS X and Windows that will repackage the .MTS files into .MOV Quicktime containers that can be accessed by MacOS X or re-encoded to a less compressed format for editing with Final Cut Pro or Premiere. We didn’t need this though, just a reliable and  quick open source workflow.

  1. The first and most important issue is to rejoin the camera split files.
    These cameras use FAT32 file systems which cannot handle individual files larger than 2GB so they split the .MTS video file into chunks. As each chunk in a continuous sequence references the other chunks these must be joined in the correct order. This is easily achieved with the cat command.
  2. The rejoined .MTS files can now be reencoded to a more manageable size using open source software such as Handbrake. We also needed to deinterlace our footage as it was shot interlaced and it would be accessed on progressive displays. This will increase the encoding time but without it any movement will look odd with visible artifacts.
  3. Finding the ‘sweet spot’ for encoding can be time consuming but in this case was important as projected text needed to be legible but the file sizes kept manageable for reasonable upload times!

 

Posted by greatbear in digitisation expertise, video tape, 0 comments