The unprecedented growth of data, its increasing importance, and business’ dependence on digital information are leading to larger and more complex information storage environments that are becoming more challenging to manage. From the perspective of data availability and protection, the information storage infrastructure is the most critical component of an overall IT infrastructure. It plays a key role in making applications work efficiently, both locally and across multiple sites. With the increasing complexity and criticality of storage, highly skilled and focused storage groups are as mission-critical as the technology being deployed.

Challenges Faced by IT/Storage Managers

These challenges are common to both large enterprises as well as SME (Small and Medium Enterprises) sectors of the industry:

• Managing storage growth
• Designing, deploying, and managing storage in a virtualized server environment
• Designing, deploying, and managing backup, recovery, and archive solutions
• Storage consolidation
• Making informed strategic/big-picture decisions
• Designing, deploying, and managing disaster recovery solutions
• Lack of skilled storage professionals
• Designing, deploying, and managing storage in a cloud computing environment
• Convincing higher management to adopt cloud
• Managing external cloud service providers

Each of these activities is on-going at various levels in each of the companies. Activities such as backup/recovery have been in practice for decades; still the professionals believe that they are not doing enough or not performing these activities well.

Complex Storage Environments

Data centers across the Americas, Europe, and Asia have deployed very similar storage solutions, including hardware and software. The sizes vary based upon business requirements, or in some cases a particular vendor may have a stronger presence in a given environment. However, on the whole, the deployed technology and challenges are very similar.

Storage Technology Segments

Storage technology deployment and its importance to the data center aligned with general market trends for each of the storage technology segments.

Each of the technology segments is unique, bringing its own specific business or operational values. For example, SAN and NAS provide connectivity options with unique functionality, while BR and replication technologies provide options for information protection against planned and unplanned outages. Technologies which enable cloud computing continue to generate significant interest.

Data Movement to Virtualized and Cloud Environments

Migrating to a highly virtualized cloud environment is a significant transformation, requiring a considerable amount of technology and business planning. Companies recognize the need for having an in-house team of professionals to lead the planning, design, and implementation of cloud and related technologies. Since cloud computing requires cross-skill expertise, IT professionals are required to have necessary knowledge across technologies that will be used in cloud infrastructure and services.

Formalized Storage Groups

Storage infrastructure is mission-critical, and a significant part of infrastructure budgets is allocated to storage-related products and services. A well-structured storage group of highly skilled professionals is key to building and maintaining high-performance, highly available storage infrastructures. Job titles and descriptions of dedicated storage professionals are evolving. With the advent of storage virtualization and cloud computing, the industry is expecting the expansion of core skills of storage professionals to include systems and networking skills.

Responsibilities

Storage groups are responsible for the overall planning, design, implementation, monitoring, managing, testing, and operation of all components in the infrastructure. Skills and processes are required to manage these tasks against expected expertise in one or more assigned “specialty” or storage technology segments.

Storage Group Skills and Performance

This is a key challenge for storage managers because it underscores the very real skills gap in their teams. Sub-optimal skills yield sub-optimal storage deployment. On the other hand, a well-skilled team will lead to higher productivity, better technology deployment and management, and optimal utilization of existing staff.

Sources for Hiring and Development

The most significant challenge faced by IT/storage managers is the shortage of skilled storage professionals in the marketplace. In fact, lack of skilled storage professionals is the most serious industry challenge. Considering the aggressive hiring requirements and plans, the lack of skilled resources becomes a serious bottleneck. There is a shortage of skilled manpower in the industry. Capable, experienced, and skilled individuals are usually not available to be hired.

Given that there exists a scarcity of certified or well-skilled storage professionals in the market, managers frequently resort to internal recruitment. Often internal recruitment involves moving an existing valuable employee who has different expertise (such as operating systems, databases, and so on) but has limited storage technology knowledge, which creates a knowledge gap in both technologies.

On-the-job training, technology vendor training, and self development by reading manuals typically cover usage and management of products and technology that is either already implemented or is in the process of being implemented. In addition, there is a need for wider and deeper training focusing on underlying technology concepts, planning, design, and management. This will enable storage professionals to independently and more efficiently design and deploy storage infrastructures that fully leverage the capabilities of all applicable storage technology segments.

Recommendations and Conclusions

From the perspective of data availability and protection, the information storage infrastructure is the most critical component of an overall IT infrastructure. It plays a key role in making applications work efficiently, both locally and across multiple sites. With the increasing complexity and criticality of storage, highly skilled and focused storage groups are as mission-critical as the technology being deployed.

Due to a lack of comprehensive storage technology education in the industry, most storage professionals have relied on on-the-job training, vendor product training, and self development.

Though probably adequate for day-to-day administration, a lack of broad and deep knowledge hampers the ability to make informed strategic decisions and to proactively plan, design, and manage storage infrastructure.

The explosion of data, its criticality, and business’ growing dependency on digital information are leading to larger and more complex information storage environments that are increasingly challenging to manage.

IT/storage managers and storage professionals across companies of all sizes face the following mission-critical challenges:

• Managing storage growth
• Designing, deploying, and managing storage in a virtualized server environment
• Designing, deploying, and managing backup, recovery, and archive solutions
• Storage consolidation
• Making informed strategic/big-picture decisions
• Designing, deploying, and managing disaster recovery solutions
• Lack of skilled storage professionals
• Designing, deploying, and managing storage in a cloud computing environment
• Convincing higher management to adopt cloud
• Managing external cloud service providers

Impact of Cloud Computing

Migrating to a highly virtualized cloud environment is a significant transformation. It requires a considerable amount of technology and business planning. Since cloud computing requires cross skilled expertise, IT professionals are required to have necessary knowledge across technologies that will be used in cloud infrastructure and services.

Complex Storage Environments

Despite the differences in industry segments and the data center size, there is a strong consistency across companies in terms of the technology deployed, storage management practices, and challenges.

Nearly all critical data is now stored on external disk storage subsystems. The average usable capacity is approximately 1.3 PB which is typically spread across multiple sites. Growth in storage requirements, larger capacity disks and subsystems, and affordable pricing have all led to large storage configurations. Storage subsystems, SANs, and backup/recovery technologies are most commonly implemented, followed by NAS, DAS, and replication technologies. Technologies such as storage virtualization and cloud (private and public) have started to emerge strongly.

Each of these storage technology segments is unique, offering their own specific business and operational value. Each requires a different set of skills for effective design and management. Lack of knowledge and expertise in a specific segment can lead to under-deployment of one or more of these technologies.

Criticality of Storage and the Need for Formalized Storage Groups

Storage infrastructure is mission-critical. Losing storage in a catastrophic situation can severely damage a customer’s business. When a disaster does occur, information on storage subsystems can be lost permanently unless a well-designed recovery mechanism is planned and implemented.

In addition to reliable equipment, a well-structured storage group of highly skilled professionals is critical to build and maintain a high-performance, high availability storage infrastructure. Storage groups are responsible for overall planning, design, implementation, monitoring, administering, managing, and operations. While the structure of the group, titles, and roles may not be standardized, responsibilities and tasks are common across companies.

The Storage Technology Knowledge Gap

Although managers prefer to hire experienced or certified storage professionals, a severe shortage of such skills in the marketplace is causing managers to resort frequently to internal recruitment. The skills gap continues to widen as organizations adopt virtualization and cloud computing.

The shortage of experienced storage professionals and the lack of storage technology education in the marketplace and in academics have restricted the growth of information storage and management functions.

Application availability is critical for business. A company’s bottom line in many ways is directly associated with the availability of business-critical applications. There is a demand to protect every bit of information, and in the event of an outage, bring critical applications back online in the least amount of time with minimal to no information loss. This brings forth a dilemma for IT organizations that they must handle with prudence as they look to consolidate storage.

Backup and recovery is, in fact, one of the key reasons for consolidating storage. By being able to account for information/storage in one location, the overall backup and recovery process can be streamlined; staggered or continuous backup and recovery policies can be employed to minimize application downtime while providing the required level of protection to business from data loss.

Backup and Recovery Objectives: At the very outset of backup and recovery consolidation, the two key elements that require consideration are the recovery-point objective (RPO) of various applications and the recovery-time objective (RTO) that the business can afford. Once RPO and RTO requirements are determined, using different technologies for various backup and recovery schemes can be deployed.

Manage/Consolidate Data that Requires Protection: Managing the total volume of data that requires protection is essential in keeping the backup and recovery environment consolidated, and it plays a significant role in minimizing the overall storage requirements and cost of data protection. Typically, organizations incur significant charges associated with backup hardware and its maintenance. There are software tools that help identify and eliminate duplicate and non-unique content from data protection environments. These tools enable organizations to retain a single instance of common information with complete security and authenticity.

The Tiered Data Protection Model: Stringent backup and recovery requirements for business productivity, corporate governance, and compliance with regulations demand faster and increased protection of information. The evolution in data protection technology made it possible to create and implement backup and recovery architectures to meet these aggressive business demands.

Consolidation with Disk Library: With the introduction of disk backup libraries, organizations can fully leverage their disk storage infrastructure. With built-in high availability features and performance, organizations can continue to leverage their existing backup and recovery procedures established for tape libraries while storing data on disk media. This can eliminate service and maintenance costs associated with managing a tape infrastructure. Where retaining some information on tape is a necessity for vaulting purposes, data can be streamed from the disk library onto tape.

Storage Management Consolidation

Storage management can be thought of as the “control tower” for storage operations. By centralizing and consolidating storage management, organizations gain increased control over their resources across multiple operation centers. This helps establish standardized and streamlined storage policies and practices, deliver higher quality of storage services, and reduce the overall storage management cost.

From a business perspective, the net result of a well-designed, consolidated storage management environment is that it improves utilization of available capital resources, increases IT staff productivity with a decline in HR costs due to fewer required resources, and decreases license costs for management tools, along with significant improvement in the quality of service.

Business Continuance and Disaster Recovery Consolidation

With the growth of business comes the need to expand IT operations across multiple geographies. When not managed properly, distributed IT operations can result in significant cost to business. In an effort to improve efficiency and reduce cost, companies may consider consolidating to fewer, centralized, and well coordinated data centers across different geographies to serve business requirements in a more consistent fashion and act as consolidated disaster recovery sites.

Planning for Storage Consolidation—High-Level Considerations

In order to realize the full benefits and ensure success of storage consolidation projects, organizations must account for and execute a number of steps before, during, and after the storage consolidation. These steps include:

Consolidation Planning and Quality of Storage Service Considerations: The first step in any storage consolidation project is to develop a well-organized plan that includes a project feasibility study based on the type of consolidation under consideration. This plan also needs to include information on the creation of a tiered storage service model based on business requirements.

Service-Level Creation: The goal of creating service plans is to better serve IT customers by leveraging available consolidated storage resources in order to deliver measurable quality of service. These service plans such as platinum, gold, and bronze, need to outline the type and tier of storage software and hardware that will be utilized.

Storage Deployment Planning: Next is to determine how storage will be consolidated without impacting ongoing business operations including IT services that will be a part of this consolidation.

Consolidated Storage Operations Planning

Once the deployment plan is articulated, the next step is to take into account the day-to-day operations of the consolidated storage infrastructure. Here we need to address:

Storage growth and scalability requirements — How storage usage will be monitored and additional resources will be provisioned

Data protection and recovery requirements — Policies that will be employed to meet aggressive recovery-point and recovery-time objectives

Information archiving requirements — Tiered storage resources that will be utilized to archive information

Disaster Recovery and Replication Requirements

To ensure non-disruptive business operations and dispersion of consistent information across multiple data centers, appropriate use of replication technologies needs planning. Organizations must consider replication technologies that offer the flexibility required in implementing appropriate replication solutions according to business requirements.

Maintenance and Repair Services Requirements

One of the major benefits of consolidating storage is to simplify necessary maintenance and repair works. There is a significant cost that businesses incur as a result of scheduled or unscheduled downtime due to maintenance activities such as data backups, storage provisioning, firmware upgrades, and repair of hardware after a failure. Organizations spend millions of dollars renewing their maintenance and service contracts with suppliers. Hence it is critical to fully take into account the reliability, availability, and serviceability (RAS) features of storage hardware to be deployed.

Operations staff and skill requirements

Another driver for consolidating storage is the need to reduce staff requirements to manage fast-growing storage infrastructures. In order to reduce operational complexity and improve staff productivity, it is imperative to employ tools that are easy to use and feature a standard, intuitive methodology for managing storage across a broad set of resources. Automation of day-to-day storage administrative tasks are essential in improving staff productivity.

Electronic business information is one of the most valuable assets of any organization and continues to grow at an unprecedented pace. As a result, storage growth is outpacing most other information infrastructure components in IT. Consolidating information onto the right storage platform delivers significant business and technical value. Storage consolidation places IT on track to respond to future business requirements in a proactive and cost-effective manner.

There are a number of ways a business can consolidate storage:

Infrastructure Consolidation with iSCSI: iSCSI storage, while still in an early adoption phase, made it possible for many organizations to consolidate direct-attached storage into a storage network. With the use of an organization’s existing IP-network infrastructure, iSCSI eliminates the need to build a Fibre Channel storage infrastructure which helps lower the deployment cost.

When considering storage hardware products, it is critical to take into account capabilities such as built-in support for Fibre Channel and iSCSI protocols to accommodate evolving business requirements of large or small companies from an investment protection perspective.

Infrastructure Consolidation with Fibre Channel SAN: Medium-to-large-size organizations that adopted Fibre Channel SAN technology some time ago continue to benefit from these deployments. The challenge today is that due to either limitations in SAN technology that existed a few years ago or due to limited initial experimental SAN deployments with additional deployments over time, companies have accumulated numerous SAN islands within a data center.

By consolidating storage to a scalable platform, organizations can achieve significant improvement in storage utilization.

Infrastructure Consolidation with NAS—File Server Storage Consolidation: The growth in business can lead to proliferation of storage resources to a point where managing these resources not only becomes difficult but also quite expensive with diminishing returns. In order to reduce total cost in implementing and operating the supporting IT infrastructure, a networked-attached storage (NAS) architecture can play a critical role in heavy file serving environments.

By consolidating file servers and storage to NAS, organizations can simplify file server and storage management, consolidate hardware footprint to save floor space, and reduce the overall cost of operations.

Information Consolidation—Optimization with Tiered Storage: Business information must be retained for many years to remain compliant with regulations and fend off lawsuits and litigations. With the ongoing growth of information, storing all business information on primary disk storage can be costly. Hence, it is important to be able to classify information based on its relevant value to business and associate the appropriate class (tier) of storage for information retention.

Tiered Storage Archiving: A properly designed tiered storage archiving solution helps organizations in several ways. First the solution classifies and maps information value to business. Next, the solution consolidates all archiveable information in a centralized location making it easier to better manage and quickly retrieve information. Finally, by being able to create a balance between information value and its repository and by migrating less-critical information to low-cost storage, organizations can reclaim their premium tier-one storage for their most-critical business information, thus driving down the overall cost of storage and keeping the environment consolidated.

Database Information Archiving: Databases, which are prone to grow very large over many years of deployment, can make use of tiered storage and effectively consolidate database and manage cost. With the use of data classification software designed specifically for all popular databases, organizations can classify their SQL, Oracle, SAP, and other database structures and store database components on different classes/tiers of storage while maintaining relational structure between all database components to make it fully functional and operational from an end user’s perspective.

Storage consolidation is the act or the ability to centralize available storage resources of an organization into a physical or virtual pool of storage resources from where storage can be effectively managed and provisioned to applications across the enterprise on an as-needed basis.

Why Consolidate Storage?

According to industry studies, the average utilization of database and application server storage is between 35 to 50 percent. Evolution in technology and the availability of network-based storage makes it possible for organizations to effectively consolidate their database and application server storage and increase utilization rates by approximately 30 percent.

Problems Addressed by Storage Consolidation

Business Solution: By consolidating storage, organizations can meet the business requirements of containing and reducing storage infrastructure costs. Organizations gain strategic competitive advantage by being able to quickly roll-out new applications and services without the need to build the associated storage infrastructure from scratch.

IT Solution: From a technology perspective, organizations are being challenged to reduce complexity and eliminate vendor lock-down. The right storage platforms enable organizations to deploy self-sustaining architectures to required applications, and, in the event of a failure, recover without impacting the business. These architectures also automate information placement and migration while applications remain online and users continue to transact business as usual.

The net effect of a well-designed storage consolidation model is that productivity improves while costs are reduced. Organizations better leverage storage and employ just-in-time methodology to procure and provision additional assets. Advanced technical strategies can be employed to appropriately protect information without incurring downtime. Customers can implement a tiered storage model and eliminate redundant data to reclaim storage resources and retain an efficient consolidated environment.

How to Effectively Consolidate Storage?—Architectural Considerations

The three high-level categories of storage consolidation include:

Infrastructure Consolidation: This type of consolidation pertains to the centralization and pooling of storage hardware.

Information Consolidation: This type of consolidation pertains to the centralization and minimization of non-unique redundant information and the placement of extracted information on the appropriate tier of storage based on its value to the business.

Operational Consolidation: Operational consolidation deals with the adoption and use of tools, policies, practices, and procedures that help centralize and consolidate storage operations.

The increased use of technology in conducting mainstream business operations has made electronic information one of the most valuable assets of any organization. This transformation in business practice also presented IT organizations with the challenge of maintaining and consistently delivering high quality of IT services to consumers.

While organizations respond to this growth in electronic information with the adoption of new storage technologies, companies are being challenged to fully materialize their return on storage investment. Much of storage resources go untapped due to a lack of diligence in technology procurement, deployment, and operational phases. As a result there is an opportunity for companies to consolidate their storage operations in order to increase resource yield and lower the total cost of operations and ownership.

What are the Storage Challenges?

As companies grow and expand business operations to tap new market opportunities, there comes the challenge of equally scaling and maintaining IT infrastructures without incurring excessive cost. Some of the key storage challenges include:

Excessive Cost — Corporate information which must be retained includes critical and non-critical business information.

Operational Cost of Data Centers — Floor space required for housing IT infrastructure is a significant operational cost.

Low Return on Investment — According to industry studies, the average utilization of storage within most organizations is approximately 35 to 50 percent of all available storage resulting in low return on storage investment.

Business Disruption — Organizations cannot afford to keep business-critical applications offline for extended periods of time due to planned or unplanned maintenance work.

Insufficient Information Backup and Recovery — The amount of electronic information continues to grow exponentially and the loss or extended unavailability of business-critical information can have drastic financial impact on a company’s bottom line.

Lack of Business Continuity — Many IT organizations lack consistent data replication policies and practices to fully protect information across multiple data centers, thus exposing business to risk.

Rogue Management — Lack of interoperability between heterogeneous storage software and hardware resources leads to unexpected storage management issues such as difficulty in migrating data between storage devices or managing multi-vendor storage using a single tool.

Lack of Access to Pertinent Information — Lack of proper reporting tools leads to inconsistent quality of storage service to users.

Complexity — Use of too many tools not only creates the pressure of training staff members but, more importantly, retaining trained staff to gain productivity and maximize investment in building a qualified storage operations team.

Compliance with Regulations

For any company, it is critical to fully account for every bit of information and store all information in a secure manner for many years in order to be compliant with various regulations.

Data can be classified as structured or unstructured based on how it is stored and managed. Structured data is organized in rows and columns in a rigidly defined format so that applications can retrieve and process it efficiently. Structured data is typically stored using a database management system (DBMS).

Data is unstructured if its elements cannot be stored in rows and columns and is therefore difficult to query and retrieve by business applications. For example, customer contacts may be stored in various forms such as sticky notes, e-mail messages, business cards, or even digital format files such as .doc, .txt, and .pdf. Due to its unstructured nature, customer contact data is difficult to retrieve using a customer relationship management application. Unstructured data may not have the required components to identify itself uniquely for any type of processing or interpretation. Businesses are primarily concerned with managing unstructured data because over 80% of enterprise data is unstructured and requires significant storage space and effort to manage.

Businesses analyze raw data in order to identify meaningful trends. On the basis of these trends, a company can plan or modify its strategy. For example, a retailer identifies customers’ preferred products and brand names by analyzing their purchase patterns and maintaining an inventory of those products.

Effective data analysis not only extends its benefits to existing businesses, but also creates the potential for new business opportunities by using the information in creative ways. Consider a job portal as an example. In order to reach a wider set of prospective employers, job seekers post their résumés on various websites offering job search facilities. These websites collect the résumés and post them on centrally accessible locations for prospective employers. In addition, companies post available positions on job search sites. Job-matching software matches keywords from résumés to keywords in job postings. In this manner, the job search engine uses data and turns it into information for employers and job seekers.

Because information is critical to the success of a business, there is an ever present concern about its availability and protection. Legal, regulatory, and contractual obligations regarding the availability and protection of data only add to these concerns. Outages in key industries, such as financial services, telecommunications, manufacturing, retail, and energy cost millions of U.S. dollars per hour.

Adapted and Excerpted from EMC’s Information Storage and Management: Storing, Managing, and Protecting Digital Information (pdf) an excerpt of which is available online. Or you can buy the complete edition here.

Provide Your Perspective in the 6^th Annual Global Study on Information Storage and Management Trends and Challenges

EMC is continuing its initiative to carry out annual research into the challenges facing the storage industry resulting from unprecedented information growth and emerging technologies such as storage virtualization, cloud computing, and big data analytics. We invite you to participate in this important survey which will identify how managers and individual professionals are responding to these challenges and help them with their planning for 2012–13 and beyond.

As a thank you for your insight and participation, the first 1000 respondents to complete the survey will receive a 2012 wall calendar (please allow 4 – 6 weeks for delivery).