Are you considering a cloud migration of all your media storage?

You are in the right place.

This article will:

• Help you gain knowledge on the most common storage types
• Provide you with a checklist of everything you need to kickstart your project

The modern-day production team is facing an array of challenges associated with traditional asset storage systems. With business drivers such as budget and capacity restraints along with digital transformation. Teams are now opting for storage solutions that focus on cost reduction, process simplification, improved retrieval performance and better accessibility and control.

All of this points to the latest storage solution related to cloud scalability.

Before we make the case for cloud storage, let’s go over some of the most commonly used media storage technologies today.

‍

Most Common Storage Types

Currently, there are five main ways companies are managing large-scale asset storage and accessibility. In the past, each of these options have been presented as effective models for unique reasons. Let’s dive in:

1. Block Storage (ie: Elastic Block Storage, SAN): High throughput for ephemeral tasks

Block Storage is a technology used to store data files on cloud-based storage environments. Also called Storage Area Networks (SAN), it is connected through fiber channel. Block storage transforms data into blocks, storing them separately with unique identifiers.

Because Block Storage breaks up data from user end-points, this allows for multiple data paths and hence quicker retrieval times. This technology relies on hard drives and SSDs at a cost that can vary from 0.006 to 0.1 USD per GB. The variation depends on the system and features it provides. It is typically used for production, post-production and heavy processing where faster access times are key.

2. File Level Storage (NAS): Ideal for internal sharing

This type of storage approach is deployed in Network Attached Storage (NAS) systems. File architecture is relying on folders in a specific order. The system itself is made of several hard drives in a physical instance that are then exposed to the network through an operating system. Depending on the backup and resilience requirement, these drives can be mounted using the RAID protocol (managing the level of duplication, to ensure a drive failure does not compromise the stored data).

This operating system layer means several users can mount a volume at the same time and access their file remotely using friendly common file protocols. It is also its main drawback as this operating system can be a single point of failure. File level storage is also highly dependent on network access speed and can cause writing and reading concurrency issues (especially when using protocol such as NFS).

3. Object Storage: Multi purpose - scalable and resilient

Object storage splits data files up into specific units called ’objects’, kept in a single storage warehouse. This storage then adds extensive metadata to a given file, removing file tree storage structure and consolidating data into a storage pool. This type of storage technology is used for archive storage and retrieval at a cost that can vary from 0.010 to 0.023 USD per GB. This is one of the basic services of public cloud providers (ie: S3 provided by AWS or Object Matrix in its cloud version - this one being also available on-prem).

Building an on-prem object storage is also possible but requires infrastructures, physical servers, some open source technologies such as Openstack, and a lot of manpower for configuration.

4. LTO (Linear Tape Open): Archives - Very competitive and low energy consumption

Linear Tape-Open, or LTO tapes, have long been considered the golden standard for archival backup and storage. This open tape format is made for high capacity and storage density. As a tape-based data system, this option offers large-scale storage capacities (i.e. 12 TB/ tape, 30 TB if compressed on LTO8) with an average 30 year shelf life. This type of storage technology is used primarily for cold archives with a very competitive price per TB.

It’s worth mentioning that these systems have an interesting paradox: Content needs to be retrieved before accessed. This means usually it will take from a few minutes to a few hours (on automated systems). It also means LTOs are very energy efficient. So on top of being very competitive, they also have a much lower carbon impact than other storage options.

5. Video Tapes

Video Tapes (Betcam, HDcam…) as a storage medium may not be as common as it was 30 years ago. However it makes the list since many companies are still dealing with them! It goes without saying that video tapes are not a great storage option for archive data. In fact they were designed to be low-cost with limited durability. The likelihood of video content loss is high. Tapes only last a certain period of time before breaking down. This storage option has traditionally been used for cold archives. If you are working on an archive migration project with video tapes it means it will require tape digitalization first. That is a very manual and costly process.

Storing in the Cloud

Simply put, cloud storage is a cloud computing model in which data is transferred and stored on remote storage. Like servers accessed over the internet or through a private network connection. Residing in an off-site storage system, this system is managed by a third-party service provider. It has the potential to store hundreds of thousands of hours of pictures, videos and other files. The migration involves moving the object and metadata associated with it, as well as performing application updates. This allows the application to know where to find the data should it need to access it in the future. It is also an environment that users trust for professionally supported back-up with secure asset transfer and access, globally.

‍

Cloud Migration Step by Step

Cloud storage offers a wide range of products and services that fits different business requirements. One such example would be long-term media retention for backup and archiving purposes. It provides a modern-day alternative to audiovisual asset storage.

The cloud allows for durable, low cost, efficient and secure asset storage and retrieval. When it comes to archives, cloud-based object storage enables better latency and recovery mechanisms. It eliminates expensive backup error cost and typical burdens related to SLAs.

So how do you start the cloud storage migration process?

1) Take inventory of your content to estimate the size of the project

• Calculate the volume
• Note all document types (photos, videos, pdfs)
• Catalog file extensions (.avi, .mp4, .mxf, .jpg, .pdf …): these may be used to create routing rules further on
• Check if these files are compressed into a single archive or accessible (this can have severe impacts when using LTO)
• Note the number of files
• From the number of files, list off all video formats and compute hours: this can be useful if transcoding is required
• Finally, build a spreadsheet with all these filenames, path, location, existing metadata, date of creation (order/priority)

2) Ensure Content Reachability 

• Do you have video tapes? 
• Where are your files stored? LTO, on-prem, s3…
• Is it available externally? Which connection (and bandwidth) can be used? (this last question will define if it is wiser to opt for a physical migration instead of using public or vpn transfer)

3) Define transfer protocols

• Depending on the project you may want to either keep content on its original location while referencing in another location for indexing and search purposes (At Moments Lab, this what we call Bring Your Own Storage)
• Transfer all the archive to a cloud storage: in this case the transfer protocol will define the duration of the project. Several possibilities exist: for a few TB and a good outbound connection, secured transfer over the internet might be more relevant. Other alternatives are possible for a bigger size of archive using an autonomous mobile archive system that can later be attached to cloud object storage

4) Project Documents for importing metadata

• Metadata Mapping (ie: custom field from system A should match the one in system B)
• Export or build a Thesaurus in a nutshell: a list of metadata values (such as keywords, tags, localisation…)
• Export or create your Thesaurus as a .CSV it’s will be the most universal format - and you don’t have to be an engineer to make it work

Once you have these elements, you have a full brief that can be used to estimate, plan, choose the right storage solution and start the migration. Moments Lab provides different storage solutions that can fit many different business requirements, whereas users need instant or cold-cost optimized and low carbon storage. Moments Lab provides the abstraction layer essential to enabling content indexation and metadata enrichment for the new library.

Do you already have a migration project in mind?

Please do not hesitate to contact our team of solution architects and cloud experts to see if we can help you!