Scale Up to the mega Data Center

While we're in the midst of a massive shift toward mega Data Centers for Cloud and co-location, I thought it would be interesting to start to explore what could possibly happen next.

In order to understand this, its incredibly important to understand why this is happening.  The economics of the data center landscape is controlled in large part by PUE (Power Usage Effectiveness, http://www.thegreengrid.org/~/media/WhitePapers/WP49-PUE%20A%20Comprehensive%20Examination%20of%20the%20Metric_v6.pdf) which introduces major variables in delivering the consumption side economics that drive toward the value chain mode of Utility (or commodity if you prefer).

Google went further and identified 19 variables that affect the efficiency of a Data Center.
http://searchdatacenter.techtarget.com/blog/Data-Center-Apparatus/The-19-variables-that-most-affect-Google-data-centers-PUE

In order to adjust the PUE variables in a meaningful way, the Data Center must be industrialized to an extraordinary extent.  At the scale of this industrialization, the variables that affect the Data Center at larger scale are much more relevant than they are in smaller sized Data Centers and thus provide a means to achieve higher levels of PUE while decreasing the work effort necessary to achieve them.

This is where the mega Data Centers have chosen to attack (manipulate, see Wardley chart below) the problems associated with Data Center efficiency.

Wardley Value Chain

But there are limits to the industrialization capabilities, particularly power delivery.  In essence, there is a limit, based on a combination of power generation location and available delivery mechanisms when we start thinking about multi-megawatt facilities.  Put another way, it becomes increasingly expensive to build Data Centers larger than the local generation (supply) and transport (delivery) of electricity.

This is effectively the complete Scale Up of the supply side for Data Centers, square footage plus electricity, that achieves the goals of optimized control of the variables of PUE.

Keeping this in mind, let's think about the groundwork for what happens after.

One relatively good attack point in continuing the Scale Up is optimizing the efficiency of electricity delivery.  Data Centers really need to get from AC (transport) to DC (computer consumption) with fewer steps, with power cost savings order of magnitude of ~ 8%.  At multi-megaWatt size, it would not be insignificant.

"Evaluating the Opportunity for DC Power in the Data Center" by Mark Murrill and B.J. Sonnenberg, Emerson Network Power

http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white%20papers/124w-dcdata-web.pdf 

where they reference

"Evaluation of 400V DC distribution in telco and data centers to improve energy efficiency", Annabelle Pratt ; Intel Corp., Hillsboro ; Pavan Kumar ; Tomm V. Aldridge

http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4448733&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D4448733

and


"400-V DC Distribution in the Data Center Gets Real" , Don Tuite

http://electronicdesign.com/power/400-v-dc-distribution-data-center-gets-real-0

But,

If the industry continues to function the way it has historically, the step after Scale Up is Scale Out.

http://www.abusedbits.com/2016/03/mega-data-centers-and-scale-out.html

Data Center Network Types

Data Center Network Types

DC Network Types Architectural

Extending on the idea presented by Greg (whom I'm a huge fan of), http://etherealmind.com/the-data-centre-network-of-networks/

The networks in a Data Center can become quite complex.  Quite a bit of this is due to security and zone considerations.

Take, for instance, the difference between a DMZ host and a true Bastion host.  One is protected (hopefully) by at minimum a firewall with some level of application awareness.  Bastion host, probably not.

This differentiation in the Data Center network has lead to regions of Data Center networking, segregating the network by macrosocopic concepts of security and utility.

Then add the concept of logic segregation on the physical infrastructure and the network gets really interesting.

These walls need to come down in favor of more microscopic, or application aligned, security capabilities.

In most cases, there is no reason that a management network and a monitoring network couldn't occupy the same logical space.  Multi-tenancy restrictions possibly being one of the restricted use cases.

There's a similar argument for production and test platform environments.  It does require that the use of these environments be prescriptive, but they are largely doing the same thing and with the possibility of placing them in a different logical plane (a VxLAN arrangement, for example) the need to purchase and build parallel hardware infrastructures may not be needed.

If that doesn't suit, just look at the way AWS is assembled.  Is there a production or test environment?

In the long run, look at the evolution of horizontal design.  Discreet building blocks are much easier to upgrade than vertical integrations.

Design it so nobody has to eat the entire apple in a single bite.

Software Defined Security not in 2016

So, I'm wondering why Software Defined Security didn't take off the way the SDx of most everything else did.

Was on a call recently with one of the big analyst organizations.  When asking a very specific question about Software Defined Security, it became very apparent that any of the promises of circa 2014 should be forgotten.

Let's take a stroll down memory lane for a refresher:

1. Simplicity
2. Automation
3. Scaleability and Flexibility
4. Cost Effectiveness
5. Increased Security

Which got me to thinking why.  I took the next 15 min at my whiteboard and came up with this.  It's not all inclusive, it's just what I could come up with quickly.

Value Chain Mapping Security Entrypoints

Mobile Device Management isn't working out so great when it places itself above the user experience.  I'm just going to come out and say, it didn't work.

Then, looking at the variety of combinations of "layers of security" applied to secure an application delivery, simplicity is simply out the door.  Some of these (identified in red), require a specialist.

Automation, while possible, covers the aspect or profile of the security control mechanism.  Putting an automation wrapper around all of it, well that's just gruesome.  The best methods available today are layered automation by mechanism with some of these still requiring zero-touch provisioning capability.

With where and how these different security functions need to be applied, scaleability needs to be in relationship to something.  What can you choose as the focus when these functions all apply at different levels and locations?

Flexibility pretty much follows scaleability.

Cost effectiveness.  Now, if you apply the cost of maintenance of devices, operating systems, security applications, application delivery frameworks, people, programmers and the application itself.  This just doesn't look like it is going to be easy to optimize.

The final item really requires the attack surface reduction to absolute minimums.  We're running new applications, on new and old software, on new and old operating systems, with new (and old) flaws and exposures. Increased Security has to be in relationship to something, like the absolute elimination of attack points in the application delivery.  Easy enough to do if you don't want the application to work, not so easy when there are zero day exploits announced all the time.  Exploits that take advantage of possibilities not even envisioned when the software was designed for any of the devices or software relationships in the path.

The contention here is that the application delivery model has to change before Software Defined Security can holistically assume the advantages applied to other constructs of the SDx model.

In any case, I don't believe 2016 or 2017 will be the year of Software Defined Security. #mapping

Mapping Exercise - Enterprise Virtualization - update

Simon Wardley http://blog.gardeviance.org keeps posting maps, exercises of maps and guidelines for maps.  Maps, maps, maps.  As he @swardley recently followed me on twitter @abusedbits, I suddenly developed an immense sense of trust and responsibility to have a look at his teachings.

In a similar fashion, but most certainly naïve to the totality of his methods, I'm mapping Enterprise Virtualization Services.  One must start somewhere....

What I was hoping for was a simplified method to create a level of prediction about what could be expected in 2016 based on where they were from my PoV in 2015.

The results as follows:

Enterprise Virtualization Value Chain Mapping 2015

As position is relative, the map is comprised of elements of Enterprise Virtualization.  These are connected where natural connections exist and placed to the best of my ability. 

I then made a decision to create a direction and magnitude vector, also being relative, but as a predictor for how those elements would advance on the map.

Network of the legacy variety is getting more complex the more virtualization is applied.  So,  up and to the left it is.  Opposing this direction, Network using SDN should get less complex as SDN functions are developed to be used with the control plane, so to the right.  Certainly not downward as it still has some level of visibility to the user.

Storage is similar to Network.  Traditional Storage will be a lot less fun to deal with at any higher densities than it is today, with a direction up and left.  Software Defined Storage will start replace traditional storage methods so, to the right.

In both cases, when the Software Defined capability completely outpaces the traditional, a wholesale shift to the +SD feature set should take place.

Open virtualization in the enterprise still needs some nurturing.  It should slide right on the map as users get more familiar with it.  It probably won't outpace more traditional virtualization methods, so only into the enterprise product space.  I also realize there are vendors with it is a product, but have a difficulty justifying it in the product space by what I've seen to date.

Platform ecosystems are tenuously on the border of products.  They still require a substantial amount of care and feeding.  I'm parking them for now.

Container technology should be one of the big movers.  Enterprise adoption is masked by the reliance on 2nd platform application types, but should be a clear win for any enterprise that makes the jump to 3rd platform, so, to the right.

Hybrid Cloud, still a manual integration process, but it should improve, so it moves to the right a smidge.

The resulting prediction, with hopeful vectors for 2016:

Enterprise Virtualization Value Chain Mapping 2016

#mapping

Update:  I picked some tidbits from the ensuing Twitter conversation.


Michael G. Nelson ‏@abusedbits  3h
@swardley Mapping smaller scale predictors in XaaS space. http://www.abusedbits.com/2016/01/mapping-exercise-enterprise.html … Wondering if you can have a look at this.

Michael G. Nelson ‏@abusedbits  3h3 hours ago

@blopeur @swardley using a perception of the previous years vectors.

swardley ‏@swardley  3h3 hours ago

@swardley Consumer Cloud Value Chain Mapping

@swardley Consumer Cloud Value Chain Areas

Ok @abusedbits, I had a look at your map. I'd counter (which is part of the point of mapping) with this.

Benoit Hudzia ‏@blopeur  3h3 hours ago

@swardley @abusedbits PS : I would have added unikernel to the map at the genesys/ custom limit. Especially linked with NVF/storage / deploy

swardley ‏@swardley  3h3 hours ago

@blopeur @abusedbits : that's the point of map, people can add / delete / debate etc.

Michael G. Nelson ‏@abusedbits  3h3 hours ago

@swardley In the fast movers, makes perfect sense. Are you confident in the more conservative Enterprise space RE adoption?

swardley ‏@swardley  3h3 hours ago

well @abusedbits, as a priority order then I'd expect ... 1) Access to utility infrastructure to trump deployment (sourcing) practice ...

swardley ‏@swardley  3h3 hours ago

2) Access to coding platform to trump concerns over if it uses VM / containers etc. ... it's a question of what is more important to user.

swardley ‏@swardley  3h3 hours ago

So @abusedbits, if someone said to me I have an non commodity / non utility baed infrastructure environment which uses containers - well ...

swardley ‏@swardley  3h3 hours ago

... provision of containers maybe a differentiator but my need is for very utility based infrastructure and this trumps.

Michael G. Nelson ‏@abusedbits  3h3 hours ago

@swardley That's the real trick there, isn't it.

swardley ‏@swardley  3h3 hours ago

@abusedbits : hence position relative to an anchor ... the anchor being user needs.

Benoit Hudzia ‏@blopeur  3h3 hours ago

@abusedbits @swardley in that case you are adding inertia in your map Which I feel is counter productive You want now not perception of now

Michael G. Nelson ‏@abusedbits  3h3 hours ago

@blopeur @swardley I was trying to create a prediction based on the map at smaller scales than WAR.

Benoit Hudzia ‏@blopeur  3h3 hours ago

@abusedbits @swardley in that case adding the spread of usage on the map would probably be useful

swardley ‏@swardley  3h3 hours ago

@blopeur @abusedbits : find duplication and bias is a big bugbear of mine.

Michael G. Nelson ‏@abusedbits  3h3 hours ago

@swardley @blopeur I've seen it before, but really "saw" it for the first time in the presentation today.

Michael G. Nelson ‏@abusedbits  2h2 hours ago

@swardley @blopeur awesome conversation

Enterprise Cloud

Let's have a look at enterprise cloud.

Enterprise Cloud Framework

The large majority of IDC's 2nd and 3rd Platform capabilities can be run and managed within the enterprise virtualization framework.

Built primarily on commodity x86 components, the financial model for this type of service closely follows the price-performance curve within the consumer virtualization industry (read "consumer cloud").  Built on vertically scaled systems, it'll vary considerably.

Cost per workload starts to flatten out at somewhere between 300 and 700 workloads, referenced to a 2 vCPU/2 GB vRAM reference unit virtual machine.  Storage as a mix of DAS (solid state and/or spinning) and archival as needed to service the application requirements.  Cost of storage is technology and size dependent.

It can be located nearly anywhere that has suitable power, cooling and data access.  This provides the business with options to utilize cloud capabilities without any of the concerns arising from a multi-tenant solution (read "consumer cloud").

Disaster recovery can be any similarly constructed system with the typical limitations of application and storage latencies along with appropriate data access to service the workload DR requirements. Ideally the "similarly constructed" DR locks in control plane and hypervisor type.

It can support, in bare metal delivery, the interconnection to applications that have requirements above the application virtualization maximums and/or alternative bare metal operating systems.

---

Where things get "interesting"….

     For Network, please review blog entry http://www.abusedbits.com/2015/11/spine-and-leaf-nodes.html for some of the concepts.  Consider that the network above the "logical rack", that supported by a ToR switching pair, really needs to have horizontal scale, it needs to be extremely robust and support significant potential East-West traffic.  It also needs to support bare metal integration and containers, in addition to the virtual hosts.

     Enterprises should be mindful of the Operations Management requirements of their enterprise cloud service, particularly as it pertains to DevOps in the management and lifecycle of the infrastructure and application.  Lower in the visibility, but enormously valuable to the enterprise, things like directory services, host security, scanning and platform overlay need to be considered. That combined with normal enterprise functions of backup, DR, monitoring and administrative access really should be looked at from a cohesive set of feature-function requirements.  Monitoring, well, un-monitored cloud is simply frightening.

     Lastly, the APIs.  Adoption of a modern infrastructure or platform initiative, which this directly relates to, is all about access to the APIs.  There are APIs for the hardware at the point of management.  APIs for the control(s) plane capabilities, and not all or well integrated without development, as the control plane functions may be separate or standalone.  APIs for any platform substrates. APIs for software defined networking.  Then, to top it all off, multi-vendor cloud services that deliver Hybrid Cloud capability across disparate systems, will have APIs.  Read this as "you'll need programmers" OR a vendor that provides these capabilities prepackaged.

New Book - Network Programmability and Automation

Several people have asked me what I thought the future of a network engineer looks like.

I didn't have a great answer for them, other than it would look different, but is still needed:  abusedbits: Modern Network Engineering

If you want to see how it'll be different, Jason Edelman, Scott S. Lowe, Matt Oswalt are authoring a book being published by O'Reilly called Network Programmability and Automation.

I've looked at this book and I believe it is a basis for the answer we're all looking for.

Network Programmability and Automation - O'Reilly Media

This book is currently Early Release, so there may be some rough edges yet.

I will comment more on it as time permits.

Episodic Intransience vs Immutable Infrastructure

Anytime you hear someone using the term Immutable Infrastructure, you are allowed to giggle, snork and/or laugh.  Servers, Networks and Storage are not immutable, they never will be.  If you write about, have recently presented on, or otherwise define, discuss or profess Immutable Infrastructure, my apologies.

Keep in mind, Immutable is the equivalent of Willie Wonka's Everlasting Gobstopper.

Immutable means that the device, construct or mechanism is unchanging or unable to change.  Abstract it as much as you like and some underlying subsystem is absolutely guaranteed to cause change.

With today's infrastructure, we are guaranteed it will physically change sometime within 7 years.  That's typical of equipment vendor product refresh.  Plus purchasing infrastructure equipment without the continuous improvement provided by product evolution cycle would result in some extremely old school equipment.  Not to mention that memory use is transient, storage is needfully consumed and the network bursts and caches as needed.  And, don't get me started on drivers.  The equipment infrastructure is not immutable.

The operating system is guaranteed to change much more rapidly, with patch release examples measured in days or weeks, the operating system infrastructure is not immutable.

Virtualization system, changes literally on demand, be it manual or automated.  So, not immutable. 

Some of the Docker aficionados talk about an immutability in the container delivery pattern.  This is tied to a repeatable, to the tune of possibly minutes,  application development process.  If it is immutable, why would it ever need to be repeatable (changing).  Just saying, it's not.

In this I tend to concur with John Willis, "no example in my 35 years of working with IT infrastructure of a system or infrastructure that is  completely immutable…"

So, what are we really talking about?  It's not immutability in any of the delivery patterns associated with modern application development.

What we are taking about is a level of intransience certainly.  The architectural pattern must be well defined, repeatable and largely inexhaustible.  It's more like Episodic Intransience, where change/improvement/SCRUM cycle delivery exist between durations of intransience.