The immediate future should bring a dramatic increase in the available storage of SAN solutions due to price drops in storage media.  Several multi-terabyte drives are now available for in the low hundreds, and this means multi-terabyte RAID 5 SAN solutions that are basically a specialized computer with multiple drives should be available in the high hundreds instead of into the thousands.  SAN solutions for the home user will become more prevalent because of this value increase and price drop thanks to iSCSI (www.intel.com/network/connectivity/resources/doc_library/white_papers/iscsi_net work_storage.pdf)

Performance issues are moot in the future because companies can simply buy more cheap SAN units to help balance the load on the existing SAN.  However, the competition with NAS units also becoming cheaper on same technology boosts the likelihood that one of the two technologies will take over in the "shared storage arena".  I personally see SAN units as the winners, since companies like Google already provide shared space for documents and emails that functions almost identically to local storage, thus eliminating the user's perception of "NAS vs. SAN".  Also, with various companies pushing fiber connectivity to homes, true personal SAN's at the ISP's local nodes may become a reality very shortly.  Imagine automatic encrypted backup to your ISP's SAN... that's a bright future and it already exists through third-party services such as carbonite (http://www.carbonite.com/).  

 The biggest pain is management of a SAN, and obviously a lot of new tools will have to be developed to make SAN management as simple as possible for it to move into the home arena.  Specifically, SAN management will have to be completely automatic and seamless. I don't want to care who has how much storage, and as the storage capacity of SAN's trends ever upwards, I may not have to.  An intelligent application can watch who's storing what how fast, and allocate space accordingly.  It makes no sense to occupy administrator time with partitioning space when the remaining space costs effectively zero to increase at will with cheap hardware.

 Server virtualization causes performance bottlenecks, but again with additional SAN capacity, these performance issues should be eliminated.  The SAN just needs to treat each virtualization as its own system in determining remaining capacity, and all performance issues should get handled just as if regular computers were connected to it. 

 Disaster recovery is easy enough, just have two differently-located SAN's that fibre backup to each other. If there are ultra-high capacity long distance fiber networks between colocation sites, which there are already, this should be a low-cost, effective means for a one-step, seamless and automatic disaster recovery alternative.  If one of the SAN's dies, then you reroute traffic to the other colocated SAN while you replace the old one's hardware.  Once the old one is replaced, then you tell the new one to mirror itself back to the new hardware, and voila, disaster averted.

 Data deduplification is dangerous and should be avoided.  For example, how do I know I really don't need multiple copies of something?  If the cost of the media goes ever downward, then why would I even care if I were taking up a tad more space then I need to?  Actually, duplcate data is an issue of management at the business level.  If I'm trying to push duplicate data to the system, I need to know why this is trying to happen and not simply prevent it from happening.  Stemming this process-wise in a business is way more productive than allowing it to happen and then just relying on my SAN to clean up my mismanaged mess afterwards.

 

We decided to jump into the SAN community about three years ago.  We chose an iSCSI SAN because of cost and ease of use.  At decision time, we contemplated an iSCSI SAN versus Fiber Channel.  What we found was an extreme difference in the way a professional administers one opposed to the other.  The FC SAN required learning new terms and concepts such as LUNDs and LUND partitioning.  This adds overhead to an already costly product.  The iSCSI, on the other hand, required no such learning curve.  Storage pools are set up and accessed the same way as local drives on a server.  Also, hardware is less costly for this type of SAN.  I believe that iSCSI SANs are winning the war on adoptibility.  We have recently upgraded our SAN with newer and higher capacity units.  Our future plans include starting to take disk-based snapshots of data for backups.  This will greatly reduce our current backup window by allowing tape backups to be written during normal work hours.  We also plan to add equipment at an off-site location to satisfy disaster recovery requirements.

Introduction - VMware's impact

My point of view is based on what I'm doing daily - consulting SAN projects in their presales phase and leading SAN installations in a storage reseller company in Czech Republic (EU). The first thing coming to my mind is most of the storage stuff today is somehow being requested for or compared to VMware:

• People request a storage system to be installed with VMware or at least should be supported to connect to VMware later.
• People ask for SAN features comparing to what VMware does with servers - high availability, hardware abstraction, central management or DR capabilities.

For administrators, VMware usually means less worries, more safety. They are happy hearing they can get similar features in the storage layer. For us, the reseller, it's good tool to compare SAN features to VMware since most people have already accepted it. Selling a synchronously mirrored array is much easier after comparing the transparent failover capability to VMotion known from virtual servers.

Simplicity - Not a Laziness

I feel simplicity is turning to be a strong attribute of SAN solutions. It's not because of lazy administrators. It's because simplicity brings more safety to people operating SANs.

There are many different systems, they are changing often and administrators are forced to learn how storage, servers, networks communicate and affect each other, globally. Not going too deeply to understand a system. It's no time for details. People in IT are not cheap and so they're often expected to sit on more seats.

In such environment, running a "traditional", complex storage system with all those hours-taking configurations might be dangerous. Yes, there is outsourcing. But there is also the cost cutting. Which one wins? The simple yet feature-full storage.

I believe simplicity brings safety to administrators' work. Not having much options how to set a system means not many ways to do it wrong. Defining a new LUN in "traditional" storage systems requires a deep understanding of chunk sizes, raid operations, caching etc. Luckily there are vendors who moved these decicions closer to simplicity.

The more we are thinking of high availability, the more we should thing of simplicity. It's hard to maintain a complex storage system always ready for failover. In many companies running some sort of HA solution, administrators are not sure their system will survive a failure. Usually it is because there are many settings and many conditions for failover to succeed. If any one fails, the whole failover operation fails.

There is another attribute extending or realizing the simplicity - automation.

Automation - Intelligent Performance Manager

In my opinion there is a big space in storage systems which can be filled with automation features. Any storage array has lots of information about the data it stores and about the traffic it serves. It can perform lots of optimization features automatically.

A brief example: There's an array with mixed SAS and SATA drives. There is a LUN with database data placed on SATA drives. The array is able to recognize the random traffic pattern and move the database data to SAS drives which can serve it faster. Or it can move just those blocks being accessed randomly, not a whole LUN. It can even adjust the chunk size or raid level after some period evaluating the traffic. On the other side - after recognizing a sequential traffic the array can communicate with the SAN client and aggregate network paths from array to client to increase throughput. Let's call it IPM - Intelligent Performance Manager ;-)

The current SAN market is being driven by software technologies that promote consolidation, high availability, disaster recover, testing, and ease of management. There other high end technologies that are also driving SAN adoption, but I see the greatest adoption rate happening in the SMB space.

FCoE is not something that I view as a second thought for an IT decision maker who needs to plan for current or future upgrades. New technologies like FCoE have to be evaluated and certified for long periods by large enterprises and is not heavily advertised enough to compell smaller organizations. FCoE is one of a handfull of shared block level technologies that are being talked about, even SAS is being looked at for SAN by smaller shops.

The biggest pain around block storage is managing growth which affects a buch of other issue such as backup strategies, performance, maintence costs and so on.

I think availabilty and replication between multiple sites are the main considerations I see when discussing SAN and virtualization.

Disaster recover is the main topic of discussion for the benefits of adopting SAN technology. Using software with direct attached storage for replication is not something that scales for large organizations.

Deduplication is something that every one needs in order to back up the amount of data growth that is out there in a timely and consolidated manner. Organizations are forced to look at this technology, but there are still a lot of questions around which method is best and who is going to be the dominant provider.

Introduction

I am not a typical enterprise storage user.  In fact, I'm not an enterprise storage user at all - I'm a consultant focused for over a decade on assisting enterprises with their storage architecture and strategy, working with businesses of all sizes, and have never worked for a vendor of storage hardware or software.  I've also written for Storage magazine, InfoStor, and others for many years, and speak at enterprise storage conferences.

This is both a blessing and a curse - I have seen far more enterprise storage environments in much more detail than most people, but I am unable to truly empathize with my corporate storage compatriots since it's not really my gear and data that I'm working with.

However, I feel that it would not be inappropriate for me to comment on this challenge. Specifically, I am would like to comment on the future of the SAN market.

How do you see the current SAN market, and where do you think it's going?

Although the commoditizing storage market would seem ripe for stagnation, the opposite is happening.  In fact, the SAN market has continued to diversify, with iSCSI recently expanding the options for storage connectivity to a new market and FC seeing rapid uptake in the virtual server arena.  Fibre Channel over Ethernet (FCoE), whether or not part of the datacenter Ethernet push, is joining virtualized I/O technology based on InfiniBand as the next wave in connectivity.  Simply put, the SAN market is not standing still.

Focus on the new world of SAN connectivity can take away from the basics, however.  Traditional switched Fibre Channel remains the healthy market leader, and the recent upgrade to 8 Gb speed has attracted customer attention.  Although they are unlikely to run out and perform a mass upgrade, I see 8 Gb FC as as much of an inevitability as 4 and 2 Gb before it.  The switch will be made, and customers will upgrade organically.

One of the big beneficiaries of 8 Gb FC will be those with virtual servers.  This new hardware is coming just as N_Port ID virtualization (NPIV) is maturing, and the combination of this with a doubling of bandwidth will make this tech extremely attractive to virtual server shops.

Even those who do not adopt 8 Gb FC connectivity in their SAN will benefit from the upshift in throughput, though, as native 8 Gb disks appear and enterprise arrays are redesigned to accept them, just as devices like the DMX-4 from EMC benefited from 4 Gb FC back ends.

As far as iSCSI goes, I think that those of us who saw the promise of this technology can finally declare victory.  Every storage device that matters offers iSCSI as a connectivity option, and most buyers are considering adopting it. It is being weighed fairly against FC, and the promise of reduced heterogeneity and cost are proving attractive to many.  In fact, it is wrong to continually compare it against FC, since many iSCSI buyers would never adopt an FC SAN due to concerns about cost or learning curve.  Indeed, much of the uptake in iSCSI comes from areas where SAN was never adopted, and iSCSI's growth can be partly attributed to these happy customers spreading the technology wider than originally intended.

One of the prime benefits we have seen of iSCSI adoption are the technologies and techniques that have come along with it.  Clustering of smaller storage systems has become a common option for scalability, and has proven itself against old modular "head/shelf" arrays.  Although the rate of adoption for security technologies like CHAP and IPsec in iSCSI remains low, they are far more common than their FC relatives.  And Microsoft's move to simplify iSCSI drivers, especially multipathing and snapshotting, has been much more successful than their proprietary equivalents.

Let us not forget humble old NAS, either.  File server consolidation to NAS filers continues to be a healthy (but less flashy) market, and NAS virtualization is on the rise as these devices proliferate.  And some in the server virtualization community are beginning to consider NFS for their (especially VMware) servers.  NAS definitely still has life and legs and will benefit from the shift to 10 Gb Ethernet just like iSCSI and FCoE.

As for the future, it is clear that most storage vendors are lining up behind FCoE.  Although true I/O virtualization, as envisioned by datacenter Ethernet and InfiniBand, may not gain traction outside the largest data centers, FCoE seems to be the inevitable next generation for massive enterprise storage.  Just about every vendor is committed to it, and the customers I have spoken to accept it as the future.  Although 8 Gb FC might delay FCoE in some cases, it will almost certainly be the predominant SAN connectivity mechanism for large block storage devices within five years.

EC2 Persistent Storage and the SAN Market

When Amazon's EC2 product came out, many companies jumped on it for good reasons. EC2 allows for fast and easy provisioning of virtual servers and bandwidth resources backed by Amazon datacenter staff. It also bills on a hourly basis, eliminating customer commitments. It was as close to on-demand cloud computing as one can get. However, one of the biggest compromises one makes when hosting on EC2 is the lack of persistent storage. Local storage on virtual instances goes away with the instance when it shuts down or gets destroyed when failures happen. And failures do occur on EC2 virtual instances occasionally, just like anything in the datacenter, real or virtual.

Today, Amazon announced persistent storage that can be mounted from an instance. This is significant mainly because it eliminates one of the biggest reasons why companies (with standard hosting requirements) aren't on EC2 yet. This new offering is in essence, an on-demand SAN solution for EC2 with snapshot backups to S3. The fear of losing data to failures is greatly reduced (or effectively eliminated) in the EC2 environment.

The SAN market has seen its lower-end market getting eroded by open-source and commodity hardware solutions from Redhat and other ISVs. For this reason, it has increasingly focused on medium-sized and higher-end markets with bigger and more powerful solutions, targeting medium-to-big enterprises, ISPs etc. Would EC2 with persistent storage cut further into the SAN market for vendors like Dell? Probably to a good extent in the SME market. With Google entering the cloud computing space, it becomes even more imminent that storage is disappearing into the cloud for majority of lower-end IT infrastructure needs. SAN vendors like Dell will probably have to push its offering further up the chain and serve customers who are competing with Amazon in the same space. The bad news is that some of these potential EC2 competitors (like Sun and IBM) are also SAN vendors. The good news is that there are many others (Google, Joyent, MediaTemple etc.) who are not.

I thought I'd ask a different group of questions:

• What worries you the most about your storage and how is your progress solving it?
• What are the most important new storage technologies?
• Is your approach to DR changing?
• Are you creating online archives and what for?
• How do you want to manage storage?
• What do you wish you had instead of what you've got?

 

I was in sales and systems engineering of storage systems for about 5 years.  My experience comes from entering a customer environment, assessing their needs (both stated and unstated), reviewing the technologies in my portfolio against those needs and against what other manufacturers could provide, and designing and implementing an architecture. 

I’ve been out of the game for about 3 years, so when I saw this case come up I spent a couple hours familiarizing myself with some of the current technologies and what the vendors are offering.  I then posed the question to some of my close friends still in industry (a couple with manufacturers, one with a VAR, and one with a consulting company) – “what has changed since I’ve been gone?”

 
Their replies, almost unanimous in view, were telling:

 
“Not much”

 
Well, what does that mean?  While I as a sales person was touting the “next big thing” every six months, the truth is that storage is a slow-moving industry - one that does not respond to bleeding edge technology very well.  iSCSI is big-time now – but when was the first product released on the market?  I believe IBM claims to have commercially installed the first one in 2001.  In many industries, entire product sets are born, grow up, and die in that time.  Do you remember the cell phone you had in 2001?

Where is the current SAN market heading?  The answer is incremental improvement.  It’s hard not to fall into the trap of believing that each new standard or technology is going to revolutionize the world.  It’s particularly hard as a manufacturer who has to decide what to invest in and how to keep all the boxes checked on the “market needs” scorecard – let alone do any innovating.   One needs to focus the most investment in the technologies with the highest leverage.  How does one identify these technologies?  That’s the subject of a dissertation, but I’ll try to address it in an additional post.  However, some short comments on the suggested technologies below might be an indication.

FCoE sounds “cool” but if it is really the future, then it’s a long way off.  I believe Howard Marks’ assessment of the situation, posted nearly a year ago, is prophetic: http://www.networkcomputing.com/showitem.jhtml?articleID=199700581 .  If it took iSCSI seven years to become a player, and it’s still not even a regular in the big games, how long will it take FCoE?  It has a little leverage, but only because it is adding value to an existing sizable investment in a company's FC infrastructure.  It's a portal, an add on.  I think Mr.  Marks is on the money.

De-dup is another.  It’s not snake oil - because it works - but it’ll never be prime time.  Certain manufacturers are touting de-dup as a way to reduce disk requirements 90% and thus enable WAN-based replication and the solution to all your disaster recovery needs.  Sound funny?  I recall working with a consulting company in Austin who were selling SANs by assessing a company’s entire disk capacity versus their used capacity and saying a SAN could help recover all their wasted space (75% ?) by connecting the drives across the enterprise - they wouldn't have to buy new disk for 24 months!  The fatal flaw was that almost all of the “wasted’ space was server-based mirrored OS drives: 18.2 GB drives (times two!) that contained a total of 6GB of data.  All that wasted space would never be accessible to the SAN because the drives were internal SCSI.  De-dup is being marketed with some of the same flaws.  De-dup’s future is minimizing device-based utilization, not enterprise-wide utilization, because de-dup is designed for a specific product, manufacturer, or standard.   Storage networking in an enterprise always contains multiple products, manufacturers, and standards.  De-dup is a useful feature to be point purchased with components, but it has no enterprise leverage.

Well, is there any good news?  NFS.  I believe Stephen Foskett also commented on this, in a way humorously opposite of my take.  “Humble” NAS is the growth engine supporting VMware servers.  If anyone has doubts about the future viability of NAS, look no further than where VMware was two, four, or six years ago.   You find significant leverage in a product or a standard that addresses or accommodates an area of nearly uncontrollable growth.  You find leverage in something that makes growth easy, and NAS with NFS support tremendously eases growth in a virtual server world.

 In fact, perhaps I should amend a statement above.  One shouldn't necessarily invest simply in the technologies that have the most leverage.  One, whether they're a manufacturer trying to develop a product, or a customer trying to solve a pain, should invest in a technology that's easy - because one that's easy is the one that will have the most leverage.  As a launching pad of discussion - what storage technologies, standards, or products give you (any contributor) the most ease in your position?  I'll take this topic up in a subsequent post addressing pains around block storage.

 

 

--------------- 

 unformatted original:

 

How do you see the current SAN market, and where do you think it's going? Feel free to highlight personal experiences. The current SAN market is clearly divided into two camps; there are those who advocate fibre channel and see it as the "pure" storage networking technology and there are the rebels promoting iSCSI. Both sides believe they own what will become (or remain) the dominant technology. Personally, I believe in the short term, the storage market will be a bit of both. In large, enterprise-class customers, fibre channel is entrenched and that's simply because FC was the only choice. These organisations have developed knowledge, processes and procedures which are reaching a level of maturity. In addition, change for change sake is a bad thing, so suggesting the wholesale re-engineering of large and complex storage SANs to new technology is unlikely to be met positively. On the opposite side, iSCSI is a great alternative for companies who choose not to or can't afford to invest in fibre channel technology. iSCSI currently also supports certain niche environments such as VMware, offering better VMware server and client support. No doubt over time even large organisations will see the merits of iSCSI and implement it in a tactical fashion where it is most appropriate. It will be up to consultants (such as myself) to expound the virtues and pitfalls of both technologies. One thing I think will not happen is that iSCSI will completely replace fibre channel. The waters are about to become muddied, however by the new upstarts of AoE and FCoE. I think AoE is niche and a curiosity but no more than that. FCoE however, has the backing of some big players, not least of which is Cisco, who clearly want to dominate the datacentre interconnect market. On the surface, FCoE seems a good play; a technology which provides a single pipe for each server. Reduction in cost and complexity. But datacentres are not that simple. Today we already deploy multiple NICs into servers, perhaps one for front-end access and one for backup, even though this traffic could concievably all go over the same cable. Why? For the same reason that pedestrians, cyclists and cars are given their own "highway"; they are different types of traffic and they don't mix well. I think we will see the same with FCoE implementations. There are many places where bottlenecks can and will occur; the I/O stack, the IP stack, the CNA buffer stack, the FCoE switch port, the FCoE ISL and so on. Cisco are already talking about technologies to mitigate these problems, but the plain fact is that large SANs (and IP networks) benefit from physical segregation where the economies of scale permit discrete physical fabrics and LANs. Storage Architects will continue to design to these principles as a way of guaranteeing performance rather than trust the performance and congestion features. It is also worth noting that FCoE technology is not coming in as a cheap alternative to fibre channel. CNA cards will be as expensive as HBA cards and the question has to be asked as to why today's HBA vendors would cannibalise their own market by selling a direct HBA replacement at lower cost. What about the long term future? I think the "physical pipe" part of the infrastructure will over time move to support the FC, IP and FCoE standards from a single fibre connection. This will implicitly also support iSCSI and iSCSI initiators will become a ubiquitous part of the O/S. This will happen not merely because a single connection can be used per server, but because commoditisation of the interconnect dictates that it will be easier to create a single HBA/CNA ASIC which supports all the protocols, much in the same way as DVD drives today support CD-R, CD+R, CD-RW, DVD-R, etc. Customers will choose whether they feel comfortable with a single or multiple physical connections as part of their infrastructure design. And for iSCSI? Clearly iSCSI has a niche while NICs are substantially cheaper than CNAs and HBAs, however if prices drop to a level where using a CNA is a no cost alternative, then iSCSI has a rocky future unless it can find new ways to differentiate itself. I will add more comments on the questions posted over the coming days.

Mission Critical iSCSI Storage Network

Storage Networking is a tricky animal...My brush with networked storage platforms started from the time we needed few hundred megabytes of shared storage for building a cluster to enable database and email consolidation in the late 90's.

The essential character of Block based network storage continues, the goal is to protect and consolidate mission critical workloads. 

Networked Storage using traditional FC-SAN's are getting more complicated in the quest for speed and functionality.

In my view, there is a need to simplify networked storage to reduce risk, decrease mean-time-to-repair and reduce costs.

Drawing from my personal experience,  More complicated SAN sub-systems and network elements are harder to understand, harder to troubleshoot and expensive to deploy....

Personally, I have moved away from SAN implementations with FC-Front end networks after attempting to use use FC-to-ISCSI routers and put up with the complexity in the nework layout and provisioning challenges.

The Approach...I took was to select medium-to-high performance native iscsi storage arrays with FC-Disk based Backend networks. Deployed the front-end network(for connecting to servers) with Redundant Gigabit Ethernet(layer 3 capable) Switches to mimic a FC network for Multi-Pathing and fault tolerence.

This allowed me to maintain the essential performance  posture(with FC disk backends) while maintaining the front-end simplicity for iscsi networking.

There is still a question performace in activities like Synchronous replication, where FC-SAN technologies are superior but these needs are becoming less acute with applications Like MS-Exchange supporting varied data replication topologies( Cluster Continous Replication-CCR, Local Continous Replication-LCR, Standby Cluster Replication-SCR). Databases(oracle,MS-SQL, MySQL, Postgres) are supporting high availability using Grid/federation, Mirorring and Master/Slave models.

Advances in Storage awareness in the modern operating systems(Windows, Linux and Solaris)  allow support for native multi-pathing for iscsi. Support for Features like VSS and VDS on Windows, GFS on Linux and ZFS on Solaris allow for less stringent application aware iscsi friendly asynchronous replication between storage systems.

Another Major Challenge is the emergence of Virtual Machines and thier impact on performance and availability of Networked Storage. The FC-Storage have inherent disadvantage in VM based environment due the multiple layers of device drivers and in-memory indirection of I/O calls.

Practical impact of on processor usage in hypervisor based VM setups using FC-SAN are evident in due to lack availability and optimization of transparent ToE type solutions for FC.

The Ability of a Virtual Machine to use raw ethernet adapters with ToE capable drivers results in little or no loss of Storage I/O performance. there is practically zero impact processor performace due to the use of networked storage.

In conclusion, With the emergence of relatively inexpensive 10 gigE switch and HBA Solutions, continued sophistication of Operating systems and applications, the time is right to start adopting iscsi for your mission critical storage networking needs.

 

 

Mirroring vs Replications in DR

As I wrote in my first insight, I'm a technician at a VAR company working with many clients on their SAN setups. We've been quite successfull selling storage solutions with synchronous mirroring to our clients in last months. The winning argument has always been the transparent failover in case of an array failure. Hearing they don't have to touch anything to keep their apps running when the array breaks is always a pleasure for the administrators attending our meeting.

As synchronous mirroring is simple in logical design, it's the best option for highly-available SAN in my opinion. Still however, I am meeting many people saying: "We're not a bank. We can afford one or two hours break and so asynchronous replication within a single server room is good enough for us." They usually expect asynchronous replication to be something cheaper to purchase, simpler to install.

To me, as the guy involved in the recovery, the idea of doing a disaster recovery of asynchronously mirrored data appears close to a nightmare. Asynchronous always means risk of loosing some data in case of array failure. During the recovery process the administrators have to check if they can recover somehow from the primary storage to rescue as most data as they can. They can't immediately redirect the storage traffic to a replica without ensuring there is no other way first. This phase if quite difficult to manage and depends heavily on administrators' skills. And that's a risk factor.

From this perspective I only see place for non-synchronous mirroring in remote replications, where the connection between primary storage and its replica is not broad enough for synchronous transfers. In all other cases, the synchronous mirroring should be the only option.

How do I see the future of HA/DR? According to my ideas of simplicity and automation I would like to see SANs better integrated with applications stored on them. I didn't say much nice words about asynchronous replication. Nevertheless I'm sure there will always be cases where async will be the only choice. For this cases storage vendors should create some generic application aware storage layer or tools to help admins pass the recovery process. Maybe even allow the recovery process to run unattended.

 

Hi everyone ...

I'd like to join the discussion, but I'm not entirely comfortable with the business model here, (e.g. significant cash rewards supplied by Dell) on several levels.

I believe I can bring a lot to the discussion (13+ years at EMC, a popular industry blog that covers this specific topic among others, access to unique perspectives, etc.), but at the same time, given the intention of this site (e.g. sponsored blogging), I'm not quite sure how it all would fit in.

If you're not familiar with my blog, it's at http://chucksblog.emc.com

Thoughts?

Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-nosh ow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; m so-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:wi dow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-f ont-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family: "Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

Why FCoE is Relevant and Where It Will Be Used

There has been a lot of discussion about Fibre Channel over Ethernet (FCoE), which has recently received major backing from the giants of the enterprise storage vendor world, but this technology remains relatively unknown to end users.  Like so many storage protocols before it, the $10,000 question is whether FCoE will take off like iSCSI or fizzle as a niche product like FCIP, DAFS, and so many others.  If it does succeed, another critical question is what this means to iSCSI, Fibre Channel, InfiniBand, and to a lesser extent AoE, expanded SAS, and other options for SAN storage.

Why FCoE Matters

With 8 Gb Fibre Channel and alternative storage solutions leveraging InfiniBand now available and iSCSI over 10 Gb Ethernet imminent, many would ask why we need another enterprise block storage medium.  In real-world applications, FCoE at 10 Gb will likely deliver roughly the same performance as 8 Gb native FC, but will be one or two years late and (initially) more expensive. Although FCoE will likely be more interoperable than Fibre Channel was in its early years, the lack of support from operating system manufacturers is also troubling.  Plus, users will soon be able to build a very similar infrastructure by mixing iSCSI and 10 Gb Ethernet, and this will include all the advantages of IP and solid support.

So why pay more for the same performance from an untested protocol?  It's all about the future, and enterprise users will go where the support is. Storage and network/SAN vendors alike are lining up solidly behind FCoE as the next-generation enterprise interconnect.  Although InfiniBand plays Betamax in this home video metaphor, with superior technology and availability, this VHS camp has all the market ammunition.  To paraphrase the (alleged) words of Bob Metcalfe, no matter what the technology looks like, the future of networking will be called Ethernet.

The biggest storage vendors are simply because they see that converging and leveraging I/O technology makes sense for them. They can swap out the physical and data link layers from Fibre Channel to Ethernet relatively easily, so the FCoE switch is an easier change than iSCSI. It is likely that they will be able to leverage commodity Ethernet hardware to reduce cost and increase their profit margins once this switch is made.  Plus, FCoE will potentially increase SAN attachment rates (and thus enterprise storage market penetration) thanks to the cost efficiency of converged network adapters (CNAs) on the server side.  From the storage side, FCoE is all good.

The drive is similar on the network side.  The era of differentiated SAN and LAN producers is over - all of the major networking and SAN vendors are repositioning themselves as next-generation I/O providers, setting up a battle in the network space to rival the mainframe shakeout of the 1980s and the PC wars of the 1990s. Converged I/O is the business model for connectivity vendors, and most are taking up the “data center Ethernet” (DCE) charge which includes FCoE as the storage protocol for virtual I/O.  iSCSI is still there in a DCE world, but FCoE takes center stage for the enterprise market.

Who Buys FCoE

It may seem strange to declare an upstart like FCoE the winner when established options like InfiniBand and iSCSI are already out in the market, but this examination of the vendors indicates that it is indeed the case.  Is this a case of the tail (vendors) wagging the dog (consumers)?  Perhaps, but they will come along willingly given the strong case presented by converged and virtualized I/O.

Enterprise buyers are ready for a next-generation SAN technology, and some are beginning to look at 8 Gb Fibre Channel.  Those that need performance will certainly buy 8 Gb FC today, but this has little bearing on the overall prospect for FCoE.  When an application requires performance and money is available, purchases will be made regardless of future strategy.

Enterprise storage and network architects are beginning to consider the implications of server consolidation and virtualization.  As they see footprint shrink thanks to compact or blade servers and server virtualization, they will begin to question the proliferation of interconnects on the back end required to keep up with the I/O demands of these super servers.  Already, virtual I/O purveyors like Xsigo are making hay in this market, and, as mentioned above, their SAN and LAN vendors are spreading the message, too.  It won’t be long before they are convinced.

Many people mistakenly assume that DCE means pushing all protocols through a single LAN, but this is not the case.  These networks will be engineered like SANs from the start, with redundant connections and transparent failover.  Although storage and network connectivity will share the same physical “pipe”, they will certainly be segregated on separate VLANs and protected with quality of service technologies.  They have to be separated – FCoE (lacking IP) will require a totally different network topology than LAN connections.

Note that, throughout this discussion, I am referring only to the large-scale enterprise data center storage market.  Smaller corporate environments have already embraced iSCSI en masse, expanding the penetration of consolidated storage concepts beyond anything Fibre Channel could ever accomplish.  And small office and home networks are beginning to embrace these concepts as well, but are relying on protocols like CIFS and AFP for file servers and may begin to look at ATA over Ethernet (AoE) and proprietary protocols like the one pushed by Zetera/NetGear instead of iSCSI.  This leaves us with a layer cake of appropriate protocols from the smallest to largest networks.  But all have one thing in common:  They are all converged and they are all carried in Ethernet packets.  Bob Metcalfe was right!

 

The current SAN market is developing rapidly, but there is such a rush to be the best emerging low-cost SAN technology targeting small to medium enterprises that it's going to get more confusing before the fog clears.

Newer switches will be needed to support FCoE, especially to go up to 10 Gbps, however if you have newer switches there may not be a need for a big initial investment. Proponents of FCoE, including many major vendors, claim FCoE (and similarly HyperSCSI) is faster since it doesn't have the overhead of TCP to worry about. However, it also limits its ease and cost of implementation, management, and ability to work over a WAN or other routed IP connection. iFCP and FCIP may gain popularity over FCoE as they are more flexible and the increased hardware layers' speeds will overcome the additional overhead needed. However, why would a small-medium business implement one of these newer FC protocols when they can just use iSCSI?  Large enterprises are probably already heavily invested in FC, and will likely stick with it.  They probably won't even bother with FCoE as there is no perceived need to transfer, though I could see some interest assuming the cost of fibre fabric equipment doesn't drop drastically by then.

I think SMB will stick with iSCSI and perhaps FC for now, large enterprise may look at FCoE especially with the big vendor support, but I would wait a little while for FCIP and iFCP to duke it out and mature before making any decisions.
If you require low-latency/high performance and already have the fiber cabling infrastructure and conduit, why turn your fast, reliable FC *over fiber* storage network into a heterogeneous, more complex environment?

Regarding deduplication, I think it's going to be huge for companies of all sizes.
Deduplication will certainly improve your IT departments RTO (recovery time objective) for longer spans of time.  You would be able to restore a 5-year old file remotely within minutes, rather than have to dig  up an old tape, deliver it to your data center, and then begin the restore. 

I'm glad companies like Data Domain and EMC/Avamar are starting to market their products to SME (small-medium enterprise).  There is increasing demand, especially in the financial industry, to store archived information for longer retention periods in order to meet new regulations.  Additional benefits such as getting rid of cumbersome tapes and easier restore capabilities, along with growing storage capacity demands and document imaging makes this technology nearly essential.  The question is what flavor is best?

Of course it depends on the needs of your company.  Hardware is almost definitely the way to go, and in the form of a hardware appliance.  I feel in-band or in-line deduplication applies to more companies, as it will avoid unnecessary disk contention outside of your backup windows.  Typically the bottleneck in a storage system is the disks and often the available disk space on those disks, so why use a post-process out-of-band dedupe system that will require much more disk I/O and capacity?  Still it depends on your needs.  If you have a very small backup window and would rather buy more disks than increase your window, then you would be more interested in out-of-band products.  There are some very interesting developments occurring with both methods, and like most technologies the one with the smallest but most scalable offering wins.

 

I don't believe in FCoE. I don't believe in storage protocol built from the first layer up. I don't believe in people needing another adapters, switches, controllers. I just see a bunch of vendors building a new playground for their old machines. Why? Maybe they missed a train of innovation. Do they need a new protocol? I don't think so. Rather they need to fight against the iSCSI newcomers who proved there was more to show than just a dual controller.

I've seen many articles celebrating Fibre Channel protocol for its all-layers-storage concept. I was celebrating too. Then we received first iSCSI array based on a hardware iSCSI implementation. We did our standard benchmarks and found the array was performing better than most of fibre arrays we had seen before. We got 96% of that cable's throughput in sequential operations. That time I lost the idea the all-storage protocol matters.

Although it might look so, I don't think there is no space for protocol innovation. Actually, I would like to see more of application support built in storage protocols. The trend I see with our custommers is building highly available applications. As the applications store their data through storage protocols, storage protocols could transfer informations about completed transactions, flushed buffers, etc., helping to keep data consistent and available on multiple places.

Will non-rotational drives create a new SAN era?

This community forced me to think of what the next big thing in storage area could be. First I thought of protocols, wrote an insight about mirroring and some feature-related things. What I forgot completely was the core of current SANs - drives.

Last year we had a great opportunity to evaluate a DRAM-based array. To me, testing new arrays and running IOmeter tests on them recalls my childhood feelings after being given a new Lego box. That time - mean the array, not the Lego - we were excited by sequential throughput speeds close to 900MBps through a pair of fibres. We changed the test to 100% random pattern and guess what - the performance was almost the same! Sure, it makes sense, no heads are seeking their blocks on disk plates, everything works at the speed of memory chips. Yet it was a feeling of crossing a bridge to a new era.

Ask yourself a question what factors you consider when setting up a new storage system. You probably don't forget to mention performance, expected traffic pattern (random/sequential), calculations of IOPs and number of drives to satisfy the database needs. Storage vendors have to create a great effort to design their controllers so they sequentionalize the data streams as much as they can, utilizing various caching and disk interface algorithms. What the world will be without these considerations? What new challenges will we face?

The first challenge might be a latency of the connection chain. The HBAs and controller interfaces create a significantly higher delays than the storage media itself. Does it matter? I'm not sure. I've met a company who couldn't accept the latency of Fibre Channel infrastructure connecting to that DRAM array. Their application did a time sensitive decisions based on data stored on the array. Let's suppose this is not a typical usage though.

The first adopters of memory based storage media are known already. They're database systems. The advantage of seek times in random patterns is great. The memory device provides performance of tens or hundreds of traditional rotational drives. Today, like anytime before, the problem is capacity being times lower and price being times higher than traditional drives. Like anytime before however, both will settle to reasonable levels. I'm an optimist.

It's a question if there will be any effective usage for non-rotational arrays other than databases. Will we ever use memory devices for sequential, say file serving applications? Now it seems like vasting money. Today's drives with powerful caching and read-ahead algorithms perform well in sequential transfers. The neck is most often at the media level or file sharing protocol design.

For the end I will try a tip for storage vendor: What if you put some memory devices as an addition to traditional drives in a single box? I don't mean just increasing storage controller's memory, rather I mean a fast disk space configurable as a LUN. Maybe you can go even further and create an intelligent feature, moving the most active random-access disk areas between rotational and memory disk spaces.

Media storage is a multi-million dollar industry across the world, and the more people keep creating, the more it is just being stored, whether needed or not. This post will reside on a server until it is copied to another, and another before being published again. This will propagate until it exists perhaps 20 times across the globe. Not much space on its own, but consider that on a global scale, every post of every blog, every piece of information existing multiple times - storage is cheap to produce and easy to sell.

Deduplication is already a success story, and can only serve to become more so. Instead of spending 100s of man hours going through each piece of data, a crawler is set to work on the filesystem, going through files and making hashes of the data, also picking out what it believes to be relevant meta-data for each file. If (and certainly when) two hashes of the same value are discovered, a note of the match is made and logged. When the crawler has been through all the data in the system, it shows the amount of file duplication, and the location of the duplicates, when an intelligent decision can be made on the necessity of the duplication, and the correct adjustments made (deletion of copies, shortcuts, etc.)

Estimates range up to a 30% saving in storage space for these solutions, which is a compelling argument for the CFO at last, and why de-duping has proven a popular technology already. It would, of course, have been easier to have started like this when filesystems were first invented, just as it would have been easier to assign a system of tagging for the entire internet in 1985, but that didn't happen, so we have to find a better way of protecting our data. Starting inside our networks doesn't seem like such a bad compromise to me.

I know of at least 2 very large networks who are already deploying deduplication technology. It seems to make sense that very large storage areas are those which will benefit the most from it. 30% of a large SAN is a more compelling argument than 30% of a hard disk for example, and the cost savings of the former are going to cover the cost of the de-duping technology. However, storage is one area where sales are increasing whilst the rest of the market slows. SANs across the world will keep on growing until there comes a point where de-duping is a simple answer with no downside. This has already arrived for some ISPs, media companies and telecoms companies. There are many more who need it, and as it becomes more popular and the price comes down, many more who will use it.

Well, I couldn't let this pass without tossing at least something in ...

I think the biggest strategic debate going on in this part of the market (e.g. block-oriented storage in SMB-type deployments) is the question of whether dedicated storage arrays will win, or servers-repurposed-as-storage will win.

Historically, small-scale storage has either been DAS (e.g. storage internal to the server), or low-end devices (SAN, NAS, iSCSI -- take your pick). 

However, things are changing quickly, courtesy of VMware.

ESX deployments are encouraging many SMBs to consider external, networked storage for the first time, primarily to use ESX's failover and load balancing features.  That's one driver.

But, at the same time, it's now very possible to run storage array software as a virtual machine, transforming DAS from multiple servers into a "virtual iSCSI SAN" or "virtual NAS".  A number of vendors are targeting this use case.

From a customer perspective, the idea of running a software-only SAN (or NAS) as a number of virtual machines should be appealing: hardware costs are reasonable, it's standard servers, deployment and management is 100% virtualized, and so on. 

Sure, a dedicated storage array would probably have advantages at larger scale ... but does all of that really matter for entry-level SAN?  And, would any purported advantages offered by a dedicated storage device be outweighed by the convenience and simplicity of a VM-only model running on standard servers?

 So, here's a question for the SMB markets Dell serves: will the future of storage be more dedicated (and optimized) external devices, or general purpose servers that are providing shared storage services for virtual machines?

I'd be interested in what others might have to say about this ...

How do you see the current SAN market, and where do you think it's going? Feel free to highlight personal experiences.

The current SAN market is going up and larger. SAN's have always been FC and block level, but with iSCSI things are starting to change. I have worked with storage for over 10 years and I have seen it grow from taking a server and adding drives to fibre channel and SCSI drives to now virtual storage, where things are driven by the software. I recently took a training class and passed the certification for a newly up and coming storage device where things are iSCSI driven. The scability of this device is awesome!  It builds in modules and you can run it over IP or iSCSI. I am seeing this type of device to begin to take over the market more than the traditional SAN consisting of, drives, switches and hosts.  You can also add modules on the fly, increase the replication on the fly and many other things. I have installed NAS, DAS and SAN and when it comes to rack space, cost and scability this device is going to step above the rest.  I will not mention the company name as I am not here to talk them up, but once the company hits the street hard, I believe you will remember the posting you saw here.

Where do you think things stand with de-duplification? Is it ready for prime-time and why or why not? Where is that space heading and what impact will it have on the rest of the market?

This is nothing but CAS storage revisted and name changed.  CAS is retrived based on its content, not its storage location so why is this being revisted again.  Yes this will be important in the back up world, but will it change storage demands, I think not.

Today, Sun announced that they were going to make a collection of storage software available as open source product.  This could be a game changer or a last-gasp effort (or something in-between) for Sun and storage.

This will get Sun a lotta love from the open source community and cast a nice glow over the company for awhile.  However, it's unlikely to change Sun's storage business economics, which are not all that compelling.  They have a decent annuity business supporting and maintaining old STK SILOs and gear, but they don't have anything new, other than ZFS, that is generating much excitement.

In the long run, Sun's open source storage software will be valuable to labs and universities and places where building your own storage can be done by creative computer science professionals with small budgets. However, it could also be the start of a new sort of standardization - something that doesn't exist in the storage industry today.  In that sense it has an element of danger - and that's worth something.

SAN and Deduplication:  
My initial thinking, which I now realize was flawed, was that the deduplication aspect of the SAN Market would become insignificant thanks to the twin factors of plummetting storage costs, increased processing speeds, and robust cloud computing storage environments like Amazon E3.   However I now see that despite these major advances several factors make deduplication a key part of any robust SAN implementation:

1) Without effective deduplication complications can arise that will corrupt or mismanage data.

2) Total data storage needs are still outstripping the reduction in storage costs.   As broadband content becomes ubiquitous video files will take up staggering amounts of storage space.

3) In addition to lowering the total storage needs of the enterprise, energy and processing needs are significantly reduced by innovative use of deduplication. 

Jon Mellon of COPAN summarizes the need for deduplication nicely:

With de-duplication, companies reduce not only the number of copies of data kept, but also the amount of data that is replicated, moved, and otherwise retained for long-term storage. An organization’s backups typically accumulate a lot of redundant data over time – some of it duplicated 10 to 30 times – but a well-architected system that employs de-duplication can reduce overall storage by a factor of 10:1 or 20:1, or even more. Altogether, that kind of efficiency has an enormous impact on every aspect of storage costs.

NetApp, a leading deduplication software vendor offers online deduplication cost calculators:
Backup Calculator |   Primary and Archive Data Calculator

Thus is appears that deduplication solutions are not only "ready for prime time" but are an increasingly important concern for your SAN implementations.   When combined with full SAN implementations vendors like NetApps offer many customers excellent storage upgrade paths without massive overhauls of existing architectures.   NetApps appears to be a key innovator in this space given their market share and their innovative solutions that connect old and new components using inexpensive SAN innovations like iSCSI.   Rather than overhaul the entire enterprise architecture it is often most cost effective and expedient to use a hybridized SAN environment where you keep the expensive legacy networking environments that are using Fiber Channels and connect them to newer components that use the far less expensive iSCSI..   

Sources and relevant linkage: 

Deduplication summary - a great post from Jon Mellon:
http://compliancegoals.com/pastissue/article.asp?art=272481&issue=2 35

Deduplication post by Heidi at Computer world:
http://blogs.computerworld.com/node/1676

Deduplication Article (dated but good):
http://www.enterprisestorageforum.com/technology/features/article.php/36 17296

NetAPP's huge success and market leadership in SAN market:
http://news.moneycentral.msn.com/ticker/article.aspx?Feed=BW&Date=2 0071210&ID=7920595&Symbol=US:NTAP

Insight from TIC member Kurt Hutchinson: 
http://arstechnica.com/news.ars/post/20080415-aconsultants-vi ew-of-the-san-market.html

Another TIC Insight on SAN Market from Mike Kramer:
http://thefutureofstorage.com/archives/36

SAN Market relinquished by big players.  Estimated value of $13 Billion:
http://masshightech.bizjournals.com/masshightech/stories/2004/06/28/st ory5.html

IBM Whitepaper - Dated but good detail on several some Storage Area Network issues:
http://www.redbooks.ibm.com/redbooks/pdfs/sg245470.pdf

Video Editing and SAN:

The explosive growth of online video has only just begun and will lead to staggering storage requirements and challenges to effective delivery of both video and regular content as the global network becomes overwhelmed with large files. 

With respect to Storage Area Networks, Video Editing offers some unique challenges, especially for implementations where multiple users across the network need high speed access to large video files as they share file edits near or in real time.

I'd suggest that SAN implementations that are specialized for video will be a fast growing and key market in the coming years, and innovation in this space is likely to become an important contributor to overall internet innovation in data compression, content delivery, and other major storage concerns.  

The following program/ integrated solutions all seek to create a robust SAN functionality for Video Editing:

Global SAN: http://www.studionetworksolutions.com/products/product.php?pci=10

Workgroup ability to edit audio/video projects and large files directly over RAID-protected storage.  High-throughput iSCSI/IP SAN.    Advantage:  Avoiding the expense of Fibre Channel storage, cards and switches.

Avid Unity: http://www.avid.com/products/unitymedianetwork/

AVID's experience in Video Editing is reflected in their high number of high end global installs.    AVID lists the followng as Unity's advantages in the high end video SAN market:

*Supports collaborative HD workflows using uncompressed HD media and high-efficiency Avid DNxHD encoded media 
Huge storage capacity of up to 40 TB with MEDIArray™ XT storage
*Supports a mix of real-time clients: Windows XP, Windows 2000, and Macintosh® OS X applications via Fibre Channel or via Gigabit Ethernet.
*Optional Avid Interplay workgroup services provide integrated asset, production, and system management, background   media transfer, and archival storage.

An Avid SAN advantage is this active online forum community:
http://community.avid.com/forums/p/13609/76665.aspx

Apple Xsan 2: http://www.apple.com/xsan/
At $999 Apples's Xsan 2 appears to be a great option for Mac users with SAN needs.   Apple lists these key features of this SAN solution for the Macintosh:

* Collaboration with simultaneous access to shared source media.
* Eliminate potential single points of failure for your mission-critical workflows.
* Pool storage resources to increase capacity, reliability, and storage utilization.

Apple's Spotlight search feature allows users to search *within* video files for content relevant to their query.  This type of video search will be increasingly important to SAN environments as the number of video files skyrockets.    I'm skeptical that this video search is very robust, rather I think it simply counts on certain types of tagging embeds rather than somehow reading the video content itself. 

Metasan: http://www.rorke.com/av/metasan.cfm

metaSAN is designed for heavy bandwidth environments where workgroup access is needed.   Examples are film and video editing, art, and healthcare applications.    metaSAN allows for simultaneously access of groups of files such as video clips, satellite imagery, medical data, and CAD files in the same fashion as if it was on a local drive.

------------

iSCSI vs Fiber Channel and your SAN Implementation:
A second related issue is the answer to the question above regarding Fiber Channel vs iSCSI implementations.    It seems likely that that iSCSI's implementations are going to gradually replace most FC implementations due to the combination of cost advantages and the fact that hybrid implementations are possible such that companies can migrate from FC to iSCSI gradually and without disruptions or risks.  

Although some IT managers may stick to their guns and continue with somewhat superior but much more expensive SAN structures, over time it seems that iSCSI will "win" and become the dominant approach.  Security issues with iSCSI appear to be unjustified due to misundertandings about the architecture which does not have the type of security risk you have with normal globally online internet applications.  Also, despite the  huge cost difference, iSCSI can offer comparable performance to Fiber Channel because the performance "bottlenecks" in most applications are not data transfer speeds - rather disk I/O issues.    ie Fiber Channel speeds are theoretically about four times faster than iSCSI, but in practical use applications won't benefit from this theoretical bandwidth advantage.