There’s an excellent discussion going on with myself, @jamesurquhart over here, some good stuff from @mthiele10 over here, and so on.

But the whole point of stability and resilience is availability. Because the fact is that availability is paramount to all, period, no exceptions. It doesn’t matter if you’re public facing like Netflix or it’s an internal application. If people can’t use it, it’s a doing nothing more than burning cash.

The idea that cloud somehow changes the equation is, in fact, completely false. Cloud doesn’t change these basic concepts – it changes how you achieve them. That’s all. Doesn’t matter if your solution is hosted or in house; if it’s down, it’s worthless. If it’s down regularly, it’s worthless. But to really understand things, first we need to understand availability and start debunking a lot of the traditional “enterprise” cruft that’s been wrong for as long as I’ve been in IT. (That’s a long time.)

RAS is not what you think it is.

This acronym has been giving me fits for ages because people continue to get it wrong at every level. (I’m equally guilty in the past, I’ll note.) They’re convinced that RAS is an acronym for: Reliability, Availability, Serviceability.

No. Just no. Take that definition, and burn it. Please. Burn every copy of it ever. It’s just plain wrong. First of all, we’re not in the 1950’s – can any of you honestly remember the last time you worried about a vacuum tube going out causing a system fault? How about the last time you had a real concern about a modern SOI method processor simply ceasing to work in a 36 month timeframe? Reliability is so outmoded in many regards, it’s not even funny.

Availability? It’s less of a bad one, but it’s still not what it gets pushed as any more. RAS is applied to hardware. Modern hardware doesn’t perform most availability functions, and hasn’t for a long time. Dual-pathing and redundant cards, sure – but that goes in the R (I’ll get to that.) Availability is primarily a function of software and not hardware. That distinction is beyond important. Even an IBM POWER 795 can’t keep your Oracle database running in a power cut unless you’ve also got PowerHA or RAC.

Serviceability is still relevant though, and accurate as well. But we need to clearly define serviceability. Specifically, it’s the ability of you to perform repairs and corrective actions without disrupting normal operations. That last part is important. And exactly why absolutely no x86 box can pass even basic RAS tests. To be able to pass basic RAS, you need to be able to perform significant corrective actions such as replacing network cards without disrupting normal operations.

Reliability is Retrograde

As I said, when’s the last time you actually worried about a CPU failing just out of the blue enough to really worry about the MTBF? We don’t even bother with doing MTBF calculations on modern processors because it’s completely unnecessary and pointless. Either they’re dead out of the box or they’re likely to keep working infinitely.

R as Reliability is a throwback to the bygone era where we had to worry about things like bad hand solder joints on the 32 ICs making up a single processor, or the bus failing because a wire wrap came loose. Excepting hard disks, the manufacturing process for a modern system long ago applied engineering to send the MTBF through the roof for most modern silicon and solder work.

I mean, let’s get right down to the hard numbers. Intel rates the MTBF of the complicated RMM2 on the S5000 family motherboards at 72 years. That’s for a single component which is in fact a complete solid state system on its own. Micron rates the P300 SSD at 2M hours MTBF – that’s 228 years before component level failure occurs (barring all other faults obviously.) Modern DRAM ICs have MTBFs so high that many manufacturers don’t even bother. This whitepaper from Smart Modular Technologies (a large OEM DIMM manufacturer) calculated the MTBF on a DDR 1GB DIMM at 63.4 years.

Reliability in the context of RAS just doesn’t matter. The parts don’t fail like they used to. So what does the R stand for? Resilience.

In a modern system, parts tend to be either good indefinitely or they fail abruptly. There’s no gray area. Resilience is the ability of the hardware to sustain through an abrupt failure, be it a faulty card or CPU or DIMM. As the aforementioned whitepaper points out, the more DIMMs you have, the greater your chance of encountering a failure – on a logarithmic scale. With modern systems, you’re talking about 16+ DIMMs – which takes your MTBF from a base >15000 to a base <1500 – a 10x greater chance of encountering a system impacting fault. (Remember that we’re talking about single-bit uncorrectable errors. You know, the ones that even ECC can’t prevent from taking down entire systems.)

So R is for Resilience. And Resilience in the simplest terms is the ability of the system to handle those faults without coming crashing down around your ears.

Availability Isn’t In Your Hardware

I want you to repeat that over and over and over and over and over again. And then keep repeating it.

Look, I don’t care what hardware you’re running on. How much of it actually does any sort of software availability on its own? The answer is “none.” Redundant modules, processors, DIMMs, disks, and so on go under the heading of Resilience. Those are there to ensure operation despite faults – not to be confused with failure modes.

Availability is about doing useful things with the hardware. That means running applications, like it or not. Go ahead, try and provide services for a business with absolutely nothing but the bare OS and no add ons. Those applications rely on more applications to make them ‘highly available’ through failing over to other hardware, automatically restarting in the event of a crash, and so on. Presuming your HA software doesn’t break. Hint: it does. Because it’s a Single Point of Failure. (Get used to hearing me rail about those. I’m just getting warmed up.) Hardware availability is “can I order it today and have it tomorrow?”

The A is for Accessibility. And no, I don’t mean Section 508. I mean being able to get at the problem to do diagnostics and effect corrective actions without disrupting normal operations any more than they’ve already been disrupted. Think about things like front mounted swappable disks, rear accessible power supplies, rapid disassembly and assembly, and you get the basics of it. The more repairs you can do without having to disconnect anything, the higher the accessibility.

By the same token, a lack of Accessibility is very bad. Even if your system is resilient, if you have to take it down to effect repairs, the benefit of the resiliency ends up virtually wiped out. If you can’t get at the problem, how are you going to fix it? Answer: you can’t without basically ripping the whole thing out and then reinstalling it.

Serviceability is not the same as Accessibility

The biggest mistake I see being made on a regular basis is people presuming that Accessibility <> Serviceability. This couldn’t be more false if it tried. They are in fact, two distinct areas. Accessibility is being able to get at the problem to diagnose it and repair it. Serviceability is being able to fix the problem with a minimum of actual disruption.

There are a fair number of parallels, but just as many differences. Does it matter that I can get at a PCIe card if I still have to shut everything down to replace it? No – since I have to shut it down anyway, it doesn’t matter that I can get at it. If I have an N+0 multiple power supply arrangement, does hot-swap give me any benefit? None whatsoever. If my disks can be swapped, but not while the system is running, what’s the point of it other than slightly faster repairs?

So to have high Serviceability, you need things like hot plug PCI-Express, hot swap disks, N+1 redundant power supplies, sliding rails with cable management and so on.

So what’s the summary of this concept exactly?

Think of it this way: R+A enables S. Resilience keeps it running, Accessibility lets you figure out what’s wrong, Serviceability makes it possible to fix without disrupting Resilience. Sort of “R+A enables S, S+R improves R.”

So what’s this have to do with Availability? Well, we’ll get to that after I first explain (in part 2) why this RAS applies to cloud and much more importantly, how to apply it to cloud. Because it really does apply to all systems one way or another, like it or not.

Stay tuned for part 2…

First of all, the most important thing to recognize is that almost any company pushing it as an OR rather than an AND is trying to sell you hardware, software or services. Period. That’s just a truism and it’s unavoidable and something they don’t like me calling them out on. Especially when I point out that they’re trying to sell you these things whether or not you actually need them. But come on – we all already know they’re going to do that.

But that’s not what we’re here to look at as much as the AND versus OR argument. There’s a lot of folks who have gone completely overboard with this idea that if you don’t do continuous deployment, you’re doing it wrong. And the simple fact of the matter is that they’re wrong. IT is not a zero sum game, nor is it strictly OR operations. Most organizations don’t want or need continuous deployment. And many organizations (e.g. Google who likes to break their infrastructure at the expense of paying customers and products) are doing it completely wrong.

So let’s start with my definition of these two terms, which arguably skews a lot closer to the dictionary than what’s being pushed:

• Stability
  The ability of a process or system to continue to operate normally despite potential error conditions or external changes; efforts to reduce the number of shocks the process or system is subjected to; and the capability of an organization to leave something in place without having detrimental impacts.
• Resilience
  The ability of a process or system to survive deliberate or accidental internal or external changes (e.g. power outages, network interruption, etcetera); the capability of an organization to adapt to changes without detrimental impacts; and the capability of a process or system to adapt to unforeseen shocks.

Sounds a lot simpler that way, doesn’t it? That’s because it really can be expressed that simply. Too many people have gotten onboard with this ridiculous idea that stability is analogous to technical debt. That somehow every organization must jump into [cloud,virtualization,buzzword] with both feet or they’re basically banging two rocks together over tin cans and string. And this is where stability comes into the picture.

Stability is not a dirty word. It is not a bad thing. It does not mean ‘stale’ or ‘obsolete.’ It is in fact, absolutely critical to running a successful business, period. Companies do not succeed because they push code to production every hour or because they sling buzzwords around like it was going out of style. (Oh, how I wish it would go out of style…) Companies succeed and fail on the basis of stability – which is not aversion to change or adaptation. It’s recognizing the simple fact that downtime is not free and that change is not always good. Customers quickly abandon unreliable services unless they have no other choice.

The other bad argument is that continuous deployment somehow contributes to resilience. It does not. Resilience is a matter of building a resilient infrastructure in all regards. That means technical and personnel. As an example: using SNA over dry pair in 2012 is not technical debt and it is not bad. However, itbecomes bad because you now only have 2 people in the whole company that can fix problems with it.

By comparison, if you had converted to TCP/IP in say, 2003, you would have been reducing stability. A total rewrite of the processes involved, the introduction of external untested factors, and putting themselves at the mercy of the Internet. Hundreds to thousands of developer hours spent writing, debugging, and testing that. So the wiser choice was always to stick with SNA – it was tested, proven reliable, and they only argument for changing it is the perception that it’s old technology.

Now fast forward, and you’ve done a from-scratch rewrite to handle TCP/IP. But you’re sending the same data – does it really make sense to change the entire user interface and entry process? There are a lot of folks who argue that ‘resilience’ means “YES! You have to! Otherwise you aren’t resilient!!” Actually, the opposite is true: if you actually have a resilient process, changing the underlying networking shouldn’t affect it.

So now that we’ve got a basic understanding of my take on it, let’s dig in.

There is an idea that resilience and stability require trade-offs in one form or another. But when we look at the example above, you should notice something. No “trade-off,” no loss of stability or resilience, and no detrimental impacts. The users in the SNA example doing data entry do not need to be retrained, since the process is the same. The developers maintaining the code simply recreated the existing business process (the data entry portion) and completely discarded the SNA aspects, instead of reinventing the process.

And this illustrates exactly why change is not always a good thing. If they had thrown everything out including the data entry portion, that means having to retrain all of the users on the new interface, as well as creating a new interface. Any change to the interface is inherently disruptive. Think about what happens when you add a field to a commonly used form. What invariably occurs is that for days or weeks, people neglect to fill out that new field. Unless you disrupted the business to spend half a day going over what is essentially a minor change. Not only that, but you’ve made it much more difficult to compare to previous copies of that form.

The same thing applies in IT. If you go from logging X, Y and Z to logging X, Y, Z, and A you’ve made it more difficult and disruptive to do data comparison. “But I know when and where I made that change,” you cry, “so I can just assume” and somebody breaks out the adage about ‘assume.’ Not only that, but even when you can clearly explain why you added A, it has a ripple effect.

When you talk about adding A and removing X and repeating this process on a daily or even weekly basis, well, you’re completely deluding yourself if you think this is a good idea. Think about how often users complain bitterly about every little change Facebook makes. Stability is about avoiding these unnecessary shocks to the system. Think of it as the process by which you carefully consider whether or not a given change is actually necessary rather than likely beneficial.

Which is why ‘being able to run as-is’ is part of it. If you can’t trust the system to operate without any changes, much less without daily or weekly changes, it’s not a stable system. Period. This is why large organizations still have large scale mainframe deployments. They’re able to count on that system, day after day, to carry out it’s tasks without fail. But these systems don’t get there without resilience – redundant hardware, failover mechanisms, error tests and corrections, and processes that have been tweaked to ensure they don’t fail. Downtime is the enemy.

Yes, we all like poking at the knobs and tweaking things and making it “better.” But you have to be prepared and able to just leave it the hell alone. If you can’t just leave the system in place, untouched, then it is not a stable system. If you cannot adjust the system in-situ to compensate for shocks, it is not resilient. Avoidance and accommodation are both necessary components.

Simply put, if you find yourself forced to do a patch due to external factors beyond your control just to keep running, it is not stable or resilient. If the shock of an API change breaks it, or an API you rely on changes daily or weekly or monthly, it’s just not resilient – you’re failing to practice effective avoidance. Without avoidance of shocks you cannot have stability, which is always paramount – and without resilience you can’t compensate for those shocks.

So, as I said; it’s an AND and not an OR. The argument that resilience and accepting downtime as ‘normal’ doesn’t fly, folks. It never should. You can’t have resilience without stability, and you can’t have stability without resilience. The idea that stability is tantamount to ‘technical debt’ is frankly, intellectually offensive. The entire idea of ‘technical debt’ is intellectually offensive. Technology is and remains about getting a task done. If you’re getting that task done reliably on a system designed in the 1980′s, that’s not technical debt – that’s stability and resilience.

And yeah, being a Sun guy for as long as I have been, well. I feel obligated to share the real facts of things, so folks aren’t reliant on spin, revisionism and unicorns.

Oracle now <3′s Fujitsu, honest!

This one is about as bald-faced as it gets. Larry doesn’t like rebranding deals, because he’s stuck with what he can get from somebody else. Larry likes to manage all the technology, to get his hands dirty – he’ll tell you that himself. He’s a technology guy.

Larry doesn’t like the Fujitsu deal. But he also can’t escape the Fujitsu deal. And it is absolutely not a new thing. In fact, the Fujitsu relationship is a very ancient tradition. See, Fujitsu is the other big half of the SPARC alliance. Oh, you thought it was a product? It’s not. It’s an acronym – Scalable Processor Architecture – and the trademark is actually owned by SPARC Internaional, Inc. The two primary members since 1986 when the SPARC V7 reference architecture was released? Sun and Fujitsu. UltraSPARC Architecture 2005? Sun and Fujitsu again.
(Of note: POWER ISA is also a similar arrangement, with the majors being IBM and Motorola, now Freescale.)

Add to this the absolute fact that Solaris customers have been 100% dependent on Fujitsu for non-T series SPARC systems (which still make up the bulk of Solaris installations worldwide) since January 2009, when Sun EOL’d the last of their UltraSPARC IV+ systems. While Jonathan “I’M NOT A TOTAL IDIOT!” Schwartz sent Sun on the failboat journey that was Rock, they leveraged a tiff with Fujitsu over support contracts to gain exclusive sale rights to Fujitsu’s SPARC64 hardware in North America under the Sun brand.

Every single M-series system is exclusively designed and built by Fujitsu. And customers running M-series chose them for a reason; they are not running applications and workloads that can be handled by the T3/T4 family’s meager performance, nor do they translate to the Exa* unicorns. And if Ellison axes the deal with Fujitsu, he immediately loses a very, very lucrative revenue stream as every single M-series customer abandons ship. The annual software maintenance on even an M5000 can easily exceed $150,000 – and that’s almost pure profit when the system only gets replaced every 3-4 years.

Oracle software running on Fujitsu hardware is new and a big deal!

Could you tell a bigger whopper? Probably not.

In fact, Fujitsu is and remains the only other company in the world which has a license to sell SPARC systems with Solaris. 9 out of 10 Solaris systems in Japan are built part and parcel by Fujitsu with an OS from Sun^WOracle. And Oracle inherited that deal, which isn’t believed to expire until 2013 at the earliest.

But here’s the thing: Sun software has been running on Fujitsu metal for over 20 years. Not “about,” not “close to,” but over. The first Fujitsu CPU to ship running SunOS? The 1986 Sun 4100 system board used in the 4/260 and 4/280, which was equipped with a Fujitsu SF9010 IU coupled to a Weitek 1164/1165 floating point unit and ran SunOS 3.2. The first actual Sun designed processor wasn’t introduced until 1992, the STP1020 AKA SuperSPARC I. (Sun has never actually fabricated their own silicon – they have always been a fabless design shop. Most CPUs were actually manufactured by Texas Instruments and TSMC, but were designed and validated by Sun.)

So it should come as absolutely no surprise to anyone at all that Oracle’s software products have been running on Fujitsu metal for decades. Three decades, actually – they added support for Unix environments in Version 3, released in 1983, which included Fujitsu metal. But the relationship goes much further than just “running on Fuji metal.”

Set your wayback machine to 1999: Oracle 8i. The i was for “Internet” before it was for “obnoxious Apple marketing gimmick.” Back then, Sun had yet to ‘open source’ Java, so you know they were close as close can be when Oracle included their native JVM with 8i. (You can probably dredge up some of the old ‘The Network is the Computer’ co-marketing materials with Sun/Oracle, too.) In parallel, Fujitsu was shipping their PRIMEpower family with the new SPARC64 GP processor, which unlike the SPARC64 II, had a simple enough MMU to run vanilla Solaris. And was one of the fastest SPARC V9′s of the day. No surprise, they ran a lot of Oracle 8i installs too.

Honestly, putting things back into the silicon on SPARC isn’t an Oracle initiative or new either. It’s Sun’s fault it got taken out in the first place! The SPARC64 II had an incredibly complex and effective memory management unit that handled most memory management tasks, even at the OS level. That was too complex for Sun, which forced HAL Software Systems to develop the SunOS derivative SPARC64/OS specific to the SPARC64 I and II. Fujitsu had to take out the advanced silicon functions in later SPARC64 family CPUs so they could run Solaris!

Oracle’s First Cloud^W^WThe First To The Cloud!

This one just made me cringe. Seriously? Ellison actually tried to pull this one off?

Look. I could go on for six weeks about Sun’s cloud attempts, dating back to the McNealy era. “The Network is the Computer” campaigns, on-premise ‘hosting’ efforts, managed hosting, Sun Compute Grid (2004ish as I recall), Sun Cloud (the 2007 reintroduction), and those are just thinking of the ones that made it out into the wild where customers could look at them.

Oracle claiming to be the first anything cloud at this point is easily the most insane and delusional thing possible. As much as I have slammed Sun over the past 5+ years, you will never catch me denying their brilliant moments over the past 20 I’ve worked with them. And Sun was shipping – not marketing, but actually shipping – the great granddaddy of cloud, Sun Grid Engine in 2000. Not a typo – 12 years ago. Oh, and they offered a free version for Solaris and Linux. And open sourced it in 2001.

12 years ago. Is it the cloud you know and [love,loathe] today? Not any more than you or I are our paleolithic ancestors. Evolution sometimes leads to relationships that are difficult to recognize, but SGE is the predecessor to all, like it or not. (While there’s predecessors to SGE, they’re all dead. SGE still yet lives. Sorta.)

That evolved into Sun Cloud, which like it or not, your modern cloud also derives from. Users uploaded their data into the Sun Cloud as a bundle which contained their data and paid for how much CPU time they used. They could also select an application from Sun’s Application Catalog (introduced in 2007) which let ISVs and developers publish their applications directly, and other users simply click to select a prepared image which included the appropriate binaries and libraries.
Larry Ellison took the axe to Sun Cloud in 2010 and yanked the plug with virtually no warning on March 7, 2010, trying to throw the whole affair down the memory hole.

“Exadata X3 1/8th rack is $200K” and “way faster.”

I shouldn’t even need to cover this one, but let’s be honest here, I need to because not everyone is an Oracle customer. And this is the biggest whopper of them all.

First of all, unless you’re a member of the Oracle Religion, you know better than to believe a word of Oracle’s benchmarks. They’ve been slapped with more fines and bans for false and misleading advertising – specifically writ benchmarks – in the past year than IBM, HP and Dell combined in the last decade. When Larry says it’s faster, what he means is “we cherry picked a workload we could do well at, then we gave our system at least 20 times as much RAM as the competitor and probably more than twice as many cores.”

That doesn’t mean Exadata isn’t faster – just that stating it as a fact is lying. Period.

That $200K will get you nothing more than a bunch of x86 boxes. Not even particularly impressive x86 boxes for the price. You can get a better deal on HP Gen8, IBM xSeries, Dell PowerEdge, and Cisco UCS than that. And if you’re buying Exadata X3-2, let’s be honest, you’re not buying it for the hardware. You’re buying it for the software pitch.

Well, based on Oracle’s current pricing sheet, to actually do anything with that Exadata X3 is going to cost you $200K (or less) up front. That’s a fair and truthful statement, to say the hardware is $200K or less. (Ignoring that they’re risking another investigation for selling to customers below GSA list.)

Based on various calculations around the web from competitor analysts, DBAs, and customers, Exadata X2 cost you a cool $1.1M for a rack of metal – and another $5-7 million dollars per year in licensing and support to use it. As Larry used to say constantly, Oracle is a software company not a hardware company. And they charge like it. And as Pythian pointed out with their excellent analysis of Exadata X2, the more storage and cores you put in it, the more the licenses you need.

Know why customers didn’t applaud ‘same price for twice the hardware’? That’s exactly the reason why – Larry could give away Exadata (but he won’t, because Oracle gives nothing for free, ever) and make it all up on the licensing. Increasing the specifications might increase performance, but it has a much bigger impact on driving Oracle’s rent-seeking^Wlicensing revenue.

I don’t even want to touch the revisionist history portion of the presentations. Suffice to say, this is barely scratching the surface. And if you do your own digging, you’ll quickly find out that Unca Larry’s unbreakable wall of bricks is just cheap paint on an empty refrigerator box.

People these days misuse and abuse the phrase ‘my hero.’ You know what a hero is? A “person who, in the opinion of others, has heroic qualities or has performed a heroic act and is regarded as a model or ideal”. Neil Armstrong has been my hero for pretty much my entire life. Often misquoted, Neil’s first words when he stepped on the moon were and have always been: “That’s one small step for “a” man, one giant leap for mankind.” Frankly, the grammar arguments are a pointless and stupid distraction. That’s what he said, and that’s what he meant.

He was just a man. The first man to set foot on another planet, but still just a man. He never pretended to be anything else. He never put on airs, pretended to be someone or something he wasn’t, or went seeking fame and glory. He didn’t go to the moon because he wanted to be famous. He did it because he earned a BS in aeronautical engineering from Purdue under the Holloway Plan. He went to the moon because it was the greatest engineering and technical challenge we had ever undertaken. He went to the moon not because of fame, not because of politics, but because he was the right man for the job.

Anyone who says Neil Armstrong was not the right man for the job, doesn’t know Neil Armstrong. He could have swapped Buzz Aldrin for fame-hound Lovell; he declined, saying that he felt Jim Lovell deserved his own command. He could have gone back to the moon. He could have had anything he wanted. Instead, he stepped aside so that others could shine.

Dozens, if not hundreds of companies sought Neil Armstrong as a spokesman – most failed. Chrysler succeeded because he believed they had a strong engineering division and wanted to help a company in financial difficulties. He never brought up the moon landing, he never brought up being an astronaut, and he became part of the research and development team.
To quote the man himself: “I’ve taken the position that, if the right situation came along, where I thought I could be of significant help .. and it would not jeopardize my honesty.” He was reputed to be a good businessman, but more importantly, an honest one.

He could have been famous; he could have parlayed that into a career in politics, in science, in anything at all he wanted. And that’s exactly what he did. He did what he wanted. When he was pressed by reporters about his lack of publicized appearances, he responded “[w]ell, I was pleased doing the things I was doing. That’s the sum and substance of it.”

He stopped giving autographs in 1994 when he found that people were selling them for large amounts of money and people were circulating forgeries. He sued Hallmark when they used his name without permission, and his barber when he tried to sell hair clippings. He settled both cases, refusing to accept any money for himself – every last cent went to charity.

He could have lived anywhere in the world. He could have made millions selling autographs and memorabilia. He could have had anything he wanted. Instead, he returned to Ohio. He lived quietly, privately, and modestly. He did what made him happy.

President Obama said he carried the aspirations of the entire United States on that fateful day in 1969; I refuse to accept that. Neil Armstrong was the living example of the aspirations, hopes, and dreams of the entire human race. He was the embodiment of the best our species has to offer. On July 21, 1969, he carried the hopes for every man, woman and child on this planet to the lunar surface. The flag he planted was American, but the footprints he left were for all mankind.

Space flight has never interested me, and probably never will. For me, it’s always been about the engineering challenges, thanks to him. The first computer system I properly engineered, I named “Eagle.” For Neil.

Neil Armstrong is my hero. He embodied all the qualities and ideals I aspire to myself. And he always will.

That’s beyond shameful, and blatantly illegal – it’s called attempted blackmail. The fact that the company had the gall to do so speaks to a total lack of ethics and governance at the organization making these threats. As to OpenStack – there is no possible way I could or would support an organization which tolerates members of any sort using threats and intimidation to influence the decision making process. And the fact that any corporate member would feel they could conduct themselves in this way calls into question governance capability and ethics of the OpenStack Foundation itself. A question which must be answered if their legitimacy is to be preserved.

And we have no idea if this is the first time or the fifth time. How many other members have they blackmailed or threatened in order to get the result they want? How many other companies are doing this? This conduct MUST be investigated, to find out just how widespread it is, and how many decisions have been enforced or changed through these unethical means.

Unless I missed a memo, the whole point of the OpenStack Foundation individual board was to provide a body representative of the community, independent from the “Platinum” and “Gold” (sponsoring companies) to prevent undue or excessive influence from companies with more money. This incident would seem to indicate otherwise.

As a result, I have no choice but to recommend and request that all individuals and corporations demand external counsel be hired to investigate this matter, whether or not other decisions have been influenced in this manner, and that all OpenStack Foundation elections and nominations be halted pending the results of the investigation. If this incident is true (and I have no reason to disbelieve Shanley’s statements,) it must be determined how widespread it has been, corrective action must be taken, and the members who have engaged in this conduct must be expelled immediately, permanently, and publicly.

UPDATE:

Folks have more or less been making a LOT of baseless accusations and slinging a lot of unjustifiable mud in Shanley’s direction because she has not publicly named the parties involved or said any more about it. First and foremost; this is in the hands of the lawyers. Why? Because there really is and was no choice in the matter. As Shanley has said: she’s cooperating with the OpenStack board as much as she’s able, and continuing to do so. That does not mean publicly naming the parties involved, nor are people entitled to that information.

Secondly, frankly, this is not a matter between the accused and Shanley, and never has been. There are four parties – Shanley Kane, the accused, OpenStack, and Shanley’s employer. Why? Because the allegations include a demand for proprietary information from her employer. That’s not a “personal” problem, even when standing for a position as an individual. That’s a corporate espionage problem. The kind of problem where if something gets leaked, stolen, or broken into? It can and will fall back on you if you don’t report it immediately. Forget losing your job – you will not only get sued out of existence, you will probably go to jail too.

Now are you going to tell me that if someone demanded you engage in corporate espionage against your employer, you wouldn’t immediately report it to HR? Hint: I worked at a company where corporate espionage was a real concern and had occurred before. Failing to report it meant you were A) fired immediately B) sued in civil court C) prosecuted criminally when possible D) the company allegedly requesting it got sued E) miscellaneous things like blacklisting to ensure you never work in the industry again. I know of absolutely no company out there that does not follow the same pattern.

So no. Nobody is entitled or owed anything beyond what Shanley has said publicly, and it would be incredibly unethical and irresponsible (and potentially slanderous) for Shanley or anyone else to go around naming names. The matter is under investigation, and in the hands of people who know how to do that the right way – as in lawyers who understand the law. Bashing her and making accusations that it’s “made up” or she’s lying because she’s not saying more is incredibly disrespectful and completely wrong; and the same goes for OpenStack. It would be absolutely unethical of them – and potentially illegal – to release the names of anybody involved until the investigation is completed. And that holds true regardless of the result.

So yes, I absolutely support the course of action OpenStack has taken and I applaud the board for their quick action. However, I am still troubled by the possibility that this is not an isolated incident – I do not know whether or not OpenStack is investigating this aspect; and obviously I will reserve judgment on the actions taken for when those actions are taken, and not before.

So knock the conspiracy theories the hell off and just be patient. Until proven otherwise, I believe that everyone involved is doing their best to resolve this in a fair and ethical manner, and to ensure it does not happen ever again. And there’s nothing more you can ask of them at this point.

So let’s talk about the outright slanderous claims and statements made by people of late.

Firstly, HP doesn’t sponsor my blog and doesn’t provide any direct funding whatsoever. In fact, the most expensive thing I have received directly from HP is a nice polo shirt to commemorate an event – estimated retail value around $35. Maybe $40 since it’s embroidered. As I’ve mentioned on my disclosure page, the events I’ve attended are organized and arranged by Ivy Worldwide. I am not privy to the details, but my understanding is that HP arranges location and speakers, and Ivy selects attendees and arranges travel. HP might request specific people, but I honestly don’t know. I’m not part of the selection process other than occasionally being asked if there are other individuals I believe would benefit from attending certain events or web presentations.

Secondly, I don’t suck up to vendors, period. Never have, never will. What I write or don’t write actually has little to no bearing on whether or not I work with someone in the future. I only write about products and services relevant to my readers that I have a genuine interest in, based strictly on technology. In fact, there are HP events I have attended which I haven’t written much about outside of Twitter, for various reasons – usually because I didn’t see much value to my readers. The policy on these events has been set in stone for over two years now – attendees are free to write whatever they please without restriction (excepting embargo dates) or even to not write about something. The only actual requirements are attending, listening, and sometimes giving feedback on the events and speakers.

Third, I don’t receive any compensation beyond the aforementioned swag, and I wouldn’t accept it if it was offered. None of us do to my knowledge. We aren’t paid to write about these events, we aren’t paid to be positive, there’s simply no money changing hands at all. It’s the same policy almost everyone else has – generally, we accept travel, accommodations and meals. And we don’t fly first class, or stay at the Four Seasons either. (That isn’t to say we’re at Motel 6 – but it is to say that I spent my last vacation at a significantly more expensive hotel.) In exchange, we pay attention, consider, and write or don’t write as we see fit.

Fourth, people still don’t seem to get it. So let me spell it out. I have been there and I have done that. And in a lot of cases, I did it years before people even thought about it. Me disagreeing with you does not mean I am an idiot, jerk, or any of the things people have called me. It means chances are very good I’ve done it or I know what I’m talking about from research. I spend pretty much all my time learning anything I can get my hands on and my entire career has been spent way past the bleeding edge on technology. End result is that I am a skeptic, I am a realist, I know what businesses are actually looking for and where their concerns are, and I am beyond tired of hyping that I can see through – much less lies, doubly so those based on flawed research. I don’t always communicate what’s false, wrong, or foolish in writing – but I also don’t say anything if I don’t know. (Nor do I waste time on obviously foolish things.)

Fifth, I still maintain the same open door policy I always have. If you think you’ve got some sort of amazing breakthrough technology that I would be interested in or my readers would be interested in? Bring it! Seriously. I don’t care what vendor you are, where you’re located, or what the product is. If you think it’s interesting? Get in touch with me via Twitter or my contact form or email or phone or smoke signals and bring it to a web presentation or an event or loan me a review unit or produce a feature length film! See point four, wherein I explain that I spend a lot of my time learning. If you think your product is sufficiently interesting, then bring it.

And sixth and most importantly; I don’t have infinite bandwidth here. Which is part of why I absolutely hate having to write posts like this – they are a giant time waste. I am in the middle of starting a business and developing software here, after all. I do not have time to read every last circle-jerk press release, track 32 websites, trawl the web, pay attention to the latest feature updates, and then write about all of it. This is my personal blog; that means I only write about things that interest me personally, and that I think merit taking the time to write about. There are plenty of other blogs out there that you can and should read for stuff like “what VMware added in Update 1″ and “How To Shave Bananas In The Cloud.” Most of the things I don’t write about aren’t because I think it sucks, or it has massive flaws, or it’s stupid – it’s because I don’t have time or sufficient information, or because other people with more bandwidth already covered it better.

Crap like this leaves a nasty taste in my mouth. Doubly so when I actually feel forced to write about it. And it’s flat out off the charts when it prevents me from writing the posts I want to. Which this has.

Frankly, getting me excited about x86 takes a lot. If you bring your A game, I’m still going to yawn at you. I have been there, and done that, since I’ve been building x86 systems for 21 years. Frankly, you’re going to have to do better than “Hey look a terabyte of RAM!” I shipped two systems with 256GB – in 2006. So what the heck did HP pull out of their bag of tricks to actually get me excited? (Hint: it’s not the faceplates. Though they are sexy.)

Also, read on – or skip to the end – to take $300 off your HP Discover registration!

Well, I could blab about Sandy Bridge-E but let’s be honest. EVERYBODY has Sandy Bridge-E features. I can ship you a Sandy Bridge-E. Today. I could go on about things I love like transparent Direct Cache Access for I/O, gobs of L2 and L3, QPI updates that finally make the bus not a joke, quad channel memory controller, etcetera. But again: these are the things absolutely everybody has. Hell, these are things that will probably go in my next workstation. So, insert epic yawn here.

But that’s not what makes a good system. The processor, frankly, is a building block. It is not a system solution. It is not the system. (And the sooner people learn this, the better.) It is one item on a list of many, many components. When you’re an ODM, that list of components is very, very long and covers everything from the CPU to the sheet metal to the wires. I should know – I’ve been there. And that’s without having to design the motherboard, too.

So let’s start with where every system starts: mechanical engineering. Which got me, well, super excited. Look, I’ve built probably north of 1,500 systems myself, by hand. Trust me when I say I know a thing or ten about what goes into a good chassis. HP’s Gen8 has pretty much the best chassis I’ve ever seen, and I had to restrain myself from hugging the folks who helped design it. It’s the first chassis to actually give me EVERYTHING I want.

First, let’s start with the base frame. It’s cold rolled steel. Near zero flex with no components installed. Strategic reinforcement to reduce weight slightly, but thick panels. Plastic faceplate plus shiny metal faceplate that goes over that, both removable. So you can tear it down to the bare metal – though not all parts are intended to be field upgradeable. That’s fine – as long as you can repair them.

Which takes us to point two – serviceability. It’s x86. It’s going to break. Period. So you’re going to have to fix it. This is where HP really made me a happy person. Everyone claims ‘tool-less.’ Nobody actually is. Except Gen8. In the hands of someone like me, a Gen8 can be unracked and torn down to every single individual component in less than 5 minutes. That is not an exaggeration – I will make this bet with every one of you right now. Me + HP Gen8 DL360, from on the workbench to individual components, under 5 minutes. Which includes removing every card, every PSU, every processor, every DIMM, the fans, and the motherboard.

Could others do it? Absolutely. Have they? Oh hell no. Try doing a motherboard swap (system planar in IBM parlance) on an xSeries – sorry, System x. I’ve done a number of them – it involves a lot of unscrewing. HP Gen8? None of that – motherboard sits on a tray. Dead board? Just swap the whole assembly. That means your turnaround time on power off to power on just went from 2 hours to 20 minutes. This is what real progress looks like in mechanical engineering on x86 systems – addressing the real problems, and real needs for service and upgrade.

Now here’s where I get really, really excited. Because this is the first real progress in x86 in, oh, about a decade. It starts with this fun little System-on-a-Chip (PSoC technically) that HP designed some years back (well technically evolved from Compaq, arguably) called “iLO” – Integrated Lights-Out. On the BabyDragon, you call it IPMI. In the G7 and earlier systems? It is the part of the system I love to hate. With the fury of a thousand suns. Because on G7 and earlier, it has limited functionality and requires very additional licensing to do anything useful with it.

Gen8 completely changes this. Well okay. Mostly changes this. First, let’s start with the little details. One, the iLO now has it’s own local flash storage. This storage holds the various drivers you need for Windows, Linux or what have you. This can be updated (more on that in another post) at pretty much any time, non-disruptively – it’s just storage space. But it also means that you don’t have to pull out some custom CD or go downloading drivers from all over the place. They’re just right there. That’s the kind of “little detail” that makes a big impact.

But, HP wasn’t content to let it stand there. They finally addressed some of my biggest gripes about iLO 3 and earlier. It’s a little HP part number called TA850AAE - “iLO Advanced License.” A $300 add-on that was pretty much absolutely mandatory to do anything useful with iLO – things like virtual media, KVM over IP, actual management beyond power cycling, and so on. In other words, it was a $300 “option” that was completely non-optional.
Well, here’s the details of what you get with iLO4 Standard and Advanced. There’s some things missing still, like Virtual Media – I still do not understand why HP has not made this standard; even the BabyDragon offers virtual media standard at no charge. But as you can see, there is no shortage of goodies available which include a lot of agent-less SNMP. Which frankly, is a pretty big deal by itself. The agent-less part – what I have to say about SNMP in general is not fit for printing. (Not HP’s fault that SNMP sucks. Well, not entirely.)

But you know what? INSIGHT ONLINE. INSIGHT ONLINE. INSIGHT ONLINE. Oh my gods it is so fantastic that it is finally in there and it is so long overdue and yes it is HP exclusive and more importantly yes it is included absolutely free. Mind, it’s a somewhat misleading name, but it’s still a fully independent call home on fail.

Look. This is something people who work on big iron not only expect but require in their systems. Same for storage systems. In fact, x86 has basically been the only platform left with no call home capability. Integrity? Call home. IBM POWER? Call home via HMC. 3Par? Call home. EVA? Call home. Sun? Well.. okay, some limited call home for some systems because Sun’s service organization just sucked. But you get the idea. Call home is not an optional capability, it is a requirement.

HP Gen8′s can and will and do call home on failures, in a fully automatic and independent fashion, which is automagically linked to your service and support contracts by HP when you buy the system. Let me put this in plain English for you: a failed DIMM just went from 2-4 hours to diagnose and open the ticket plus another 4 to wait for callback to schedule, to HP’s onsite service tech calling you to schedule replacement before you even find the fault. By the way, did I mention it’s a standard included feature as part of your base purchase warranty?

Because hey by the way  Insight Online is completely free, included, zero cost, no additional work required for any and every single Gen8 customer without exception. You may want to read that line two or three times, because my friend, it is completely and totally true. And very exciting. Bought 5 servers? You’re covered. Bought 500? You’re covered. (And yes, if you prefer, you can turn it off.)

And lest you doubt this is HP exclusive; Dell offers zero call home capability. None at all. Their idea of “Proactive” maintenance is firmware updates and ‘predictable costs’. Yeah, no.
IBM offers it, sort of. I’ve half-implemented it. First, you’re going to need IBM Systems Director – System x Standard Edition. That’s going to cost you. Then you’re going to need a server to put Systems Director on. Then you’re going to need SDSM – Systems Director and Support Manager – which you guessed it, is more money. Then you’re going to need to install agents on every system you want to monitor. Then you’ll also need either the Lightpath Upgrade Kit (only for select M4s) or you’ll need a separate card or additional SDSM license. Oh and additional service and support contracts for each system. Which then have to be manually associated. And let’s not forget that SDSM is so bad, that the POWER version was withdrawn from market – to a great deal of customer rejoicing.

So yes, it is an HP exclusive feature. One that genuinely, truly excites me. I’ve been excited, but very skeptical, since February when it was initially announced. Yeah, nobody out there is going to deny the Gen8 launch was flashy, and the boxes themselves are much better looking than the G7 and prior. But these are expected things. I also expected that Insight Online was going to be an extra cost feature – maybe not an expensive one, but something that was only included with an extended support contract or additional license. I had a lot – and I do mean a lot – of questions about it.

And every answer that HP has given, has made me a real believer that the Gen8 is the first real game changer in x86 in nearly a decade. The last true game changer was the Iwill H8501 – a quad socket Opteron system for OEMs introduced in 2004 with the capability of up to 128GB of RAM. At a time when having 4GB in an x86 desktop was considered insane. Let’s break down the answers that made me a believer:

• What does it cost? – Nothing. It is included standard at no additional charge with all Gen8 systems.
• What additional hardware / software does it require? – None. It runs on the iLO itself, fully independently.
• What information does it send? – System identifying information, fault identification, minimum relevant troubleshooting data.
• What errors can it handle? – Any and all hardware faults including CPU, DIMM, IO cards and disk.
• Does HP need VPN access or similar? – No. HP does not require ANY access to your network.
• Can it be disabled in secure environments? – Absolutely; it can be fully disabled.
• Can secure/large environments set up a local controller? – “Not yet, but it is being worked on.”

So yeah. This really is a genuinely big deal. And there’s still more - like System Update Manager (SUM) turning into pure awesome. No, seriously. But that’s for another post – because you really will love it. I promise. They took 90% of the stuff that made software like it absolutely awful out of SUM.

So yeah. I EXCITED. That HP and Ivy Worldwide sent me to Houston for this event has less than zero bearing on that excitement level, too. I’ve been excited since the announcement earlier this year because I saw serious potential with Gen8 and Insight Online. Potential doesn’t always equate to a good product, but HP delivered here. It’s not perfect but HP still has a very clear winner here.

Oh, and take $300 off your HP Discover registration!

And thanks to HP and Ivy Worldwide, I get to offer you a special deal. Budget tight? Looking to save a few dollars? How’s $300 off your registration for HP Discover 2012 sound? (Besides “freaking awesome!”) And lest you think it’s just HP, here’s the list of other companies exhibiting at Discover. So yeah, it ain’t just HP! (I’ll be talking more about why you should go to Discover later in the week. Think of it as “VMworld in June” as a bit of a hint.)

So how do you get $300 (that’s just over 20% for individuals!) off your registration? Simple! First, you have to register via this link here. Super important that you use that link, because otherwise your discount won’t work. Click the Register Now button. Now select that you have received a registration code and enter “BLOG” – minus quotes, but in all caps. If it asks you for a discount code, enter “BLOG” there as well.

Extra Disclaimer Required: Ivy Worldwide and HP are running a contest for a bunch of us (17 in total!) to help drive Discover registration. If I get the most registrations through my link, they’ll ship me off to the HP event of my choice (within reason of course.) Somehow, don’t think I’m going to win that. There’s also a random drawing for a new laptop, so, I’ll let you know if I win that. (With my luck, yeah, probably not.)
But you know what? I still want people to go to Discover. These kind of events are invaluable for everyone on so many levels, and if I can make it possible for you to catch up with that Twitter friend from the other side of the world or enable you to talk to someone who can tell you how to fix that stubborn problem at work? I’m all for it. So there you go – $300 off. What’re you waiting for? Click!

This ruling is… a very bad idea. It is too broad. It takes away protections which are absolutely vital to driving innovation and invention. (And no, this is not contrary.) To describe the ruling as overly broad in a dangerous way is much like calling the ocean damp. And yes, I really do know what I’m saying and talking about here.

EDIT 05/25/2012: Hello all 10 of you from the cowardly post on G+; I guess a certain unjustifiably arrogant person figured I wouldn’t notice? But, I’ve invoked my Zero Tolerance For Those Wishing Me Harm policy. So consider the source and feel free to engage in reasonable – not religious – debate.

Everyone inevitably is going to draw parallels to Oracle v. Google – honestly, it’s irrelevant and it matters absolutely none at all here. Not one bit. Related? Sure. But relevant to this discussion? Not at all.

Here’s the problem: if you say “the expression of those ideas and principles is protected by copyright” but if you also say that “the functionality of a computer program nor the programming language and the format of data files used … do not constitute a form of expression” in the same breath? The expression of ideas and principles is not protected by copyright!

People these days just do not seem to get how software development works and why being able to copyright an API is such a critical protection to businesses of all sizes. Not just Microsoft, but every startup out there. First and foremost; there are hundreds and thousands of licenses and copyrights available for businesses to use. I can GPL my API and throw full documentation out into the world, or I can say ‘nope, mine’ and protect the details zealously. For innovation to thrive, both options must be available. Not just one or the other.

In SAS v WPL, it was determined that WPL had created a re-implementation of the SAS Language, using documentation from SAS (which is okay,) but also using a version of the software which was licensed for personal and educational use (which is very wrong.) Why? Simply put, WPL is a commercial entity. Their use of the personal/educational license for commercial work was a major violation of the licensing right out of the gate – and should have been fatal to their case by itself. Their WPS product costs many thousands and thousands of dollars (it runs on z/OS!), yet they used a version which was restricted to personal and educational use.

When SAS learned of this, they brought a suit alleging that WPL had illegally copied manuals and components, and infringed their copyright. Yes, well, this was about the most horrible test case I could imagine too. SAS is a programming language, not really a product. That’s what lead to this disasterpiece of a ruling.

First of all, trying to copyright a programming language in this fashion is absolutely ridiculous and pointless. It’s a programming language – you have to provide all the documentation and functionality to users. Once it is out there, there is no taking it back! Just ask IBM when they fought Phoenix over the PC BIOS. It was just a terrible, horrible idea to make it the test case from the outset.

The end result? “[N]either the functionality of a computer program nor the programming language and the format of data files used in a computer program in order to exploit certain of its functions constitute a form of expression.” That’s a direct quote.

Look. Copyright has a role to play in innovation and openness, whether or not that court wants to believe it. And being able to protect components like, say, the data files that your program operates on? Are a fundamental level of protection, period. If a company cannot even protect a file format in the most basic of ways, then they will be forced to use horrible ideas to protect it – things like per-machine keyed encryption. (These are not new techniques, and nobody subjected to them, even remotely likes them.) It doesn’t just stifle innovation, it grabs it by the throat, choking it while shaking vigorously.

Forget programming languages; assume that these are not copyrightable if they are documented publicly. If I create “ZYX The Compiler” and the ZYX language, then sell ZYX The Compiler with a complete documentation set of the primitives – a necessity for use – the only thing I have any hope of effectively copyrighting is the compiler itself. The language is out there. Done deal.

But let’s say I created a programming language which is internal to another product I make. And that I do not release this programming language’s documentation outside my company, I do not license a compiler or interpreter, it’s strictly an internal use product. It’s a very special piece of the puzzle that we slaved over and created just for this product.

If someone comes along, and in the course of reverse engineering the product legally (that’s legal too under this ruling, by the way,) discovers my programming language. They then implement it partially and start selling a compiler to other people. Under this ruling? I can’t do anything about that. Even though they only found the language through reverse engineering, they have no license for it, and they’re charging others for something we created.

If you don’t see that as a problem to innovation, well, I guess you’re a subscriber to the insanity of “free spending.” It didn’t work then, and it won’t work now.

If an API cannot be copyrighted, then a license also cannot be enforced. Licenses are based on an organization or person owning the copyright to the relevant licensed product. Because I cannot copyright the API, only the implementation, licensing has been rendered completely irrelevant and pointless. Not ‘opens the door to piracy en masse’ pointless, but more like ‘I just took VMware’s entire undocumented API set, posted it on the Internet, and made a VMware to HyperV converter that I charge $5,000 for without paying one red cent.’ That pointless.

That VMware’s license says I may not reverse engineer the code, blah blah DMCA, doesn’t matter in the least. That license is now rendered moot and invalid in the EU. Even though it’s an internal, undocumented, private API set which requires disassembly to discover? I can go right ahead and do that, and in the EU, VMware has absolutely no recourse. Under this ruling, I didn’t do anything illegal or violate their copyright, because they can’t legally copyright those APIs. Any of them. Which means they also cannot enforce any license, because said license is based on copyright.

It’s just too broad a ruling and it’s very bad for business and innovation. Flooding the market with clones is not good for innovation or good for business – it just creates a huge glut of identical organizations doing the same thing, resulting in all of them blowing up ultimately. (Again, this happened during the dot-bomb days too.) If I sell my software in the EU, I now can virtually guarantee that it will be reverse engineered and components will be reimplemented by shady software houses. And I will have no recourse to prevent them from stealing my customers, selling substandard software to customers, or any other number of issues that will interfere with my ability to run a successful business.

As soon as that happens, I can rest assured I will end up in a price war – after all, we’ve already done all the heavy lifting of developing the API, they’re just reimplementing it from our work – that I cannot possibly win. The only way I can win is to either burn cash at a dangerous rate to reinvent the wheel we already built, or to stop innovating and spend all my time threatening customers with the equivalent of excommunication if they use the knockoff software.

The other more glaringly ignorant part of the court’s ruling is the simple fact that APIs are not always consistent. In fact, they’re rarely consistent. Record inserts on Oracle 11g are performed nothing like those on PostgreSQL. Yet in the view of the court, they are the exact same, despite the formatting, syntax, and commands being unique. Neither Oracle nor PostgreSQL claim to copyright what the API does, either – only the specifics of the API itself. The API in and of itself is an expression of effort and ideas, in that it had to be created and mapped to functions – this is not nearly as easy as people think it is.

As I said; the point of copyright is not always to come along and say “YOU CAN’T DO THAT!” It’s to say “hey, this is our property, so you need to obey the license terms.” This ruling basically takes that and says “license? Ha. Null and void.”

Just a terrible ruling from a terrible test case.

If you follow me on Twitter, you probably noticed I got rather pissed off at someone the other night. Justifiably so. As @wilw says, “Don’t be a dick.” Well. A lot of people have been failing utterly. Not just one person – a lot. And I’m tired of feeling like a doormat and/or invisible.

Why? Because a lot of people have been saying things that are extremely offensive, whether or not they realize it. In this case, another person quite literally said that I, as in me personally, pretty much don’t deserve to live. Or to have any quality of life. While spreading ignorant, hurtful lies about the reality of the country we live in and how insurance works.

And yes, I’ve gone back and rewritten this post. Because the personal aspect of it, in hindsight, obliterated the message I actually wanted to send. Pobody’s nerfect, most especially of all not me. Especially when upset. And that’s on me, and I’m sorry.

Let me explain a social norm here. Maybe it isn’t a social norm anymore, but I happen to believe that saying someone should die is completely unacceptable. No matter who they are. People have been trying to cloak this in quasi-religious terms like “these are my core political beliefs.” And what if I said my core political belief was that your favorite person should get trampled by stampeding boars? Does that make it okay? No – saying you believe someone should die is never okay. Saying someone is a friend and in the same breath saying they should be left to die for lack of medical care is in fact, the same as saying you believe they should die.

See, without some sort of meaningful health care reform, which includes forcing insurers to do things they don’t want or gods forbid, actually fixing this completely broken “system,“ I WILL DIE. This is not “might die,” this is not “could die,” this is not “will die sooner.” I will suffer an excruciating, ugly, horrifying death I would not wish on my worst enemy. Full stop.

And it’s not just me. It’s millions of people. It’s some of my closest friends too. So, yeah, it’s personal. It’s always going to be personal for me. Because this is what I have to live with and worry about every day.

And to have the absolute gall to imply that this is a result of us “not working hard enough” or failing to “fix our situations” or blaming us in any way. That is the most insulting, most ignorant, most hateful, and most hurtful lie there is. It is complete bullshit, and shame on anyone who would spread this absolutely false information. And how dare anyone accuse us of not working hard enough. I work my ass off – far more than most people have probably ever done in their lives. I log on average 16 hours a day, working to get Tiger South launched. 16 hours a day, 7 days a week. Prior to that? I couldn’t take a vacation for 12 YEARS and the last one I had was in 2008, excepting one day at VMworld in 2010.

When I apply for health insurance, do you know what happens? They say no. Do you know why they say no? Because I’m not profitable and because I require medical care. And still do. If I’m lucky I will get to pay >8x the premiums YOU do and continually have claims denied. So unless you’re in my boat, you most likely have absolutely no business saying one word about the problems with health care in America. Ever. Watching a bunch of talking heads on TV does not make you informed – it makes you a gullible idiot, their favorite audience. On both sides of the debate. There is no truth from either side – they’re all lying to score points with their quasi-religious political base.

This is not an issue where people “need to shut up and pull themselves up by their bootstraps.” It’s quasi-religious political rhetoric that caters to the ignorant extremist bases the politicians have cultivated. Again, on BOTH SIDES. There are no innocents here, just corrupt politicians lying to everyone. So you left-wingers need to take notice too.

It costs me over $300 a month in medications just to be able to pass for functional. That number will go up in this losing battle. There is not one day I am not in extreme pain, and I will never get to do the things I enjoy most in life ever again. Those of you who read this post, will all outlive me. Even if nothing else goes wrong – insert bitter laugh here – my life expectancy is 57 years, 60 at the extreme. I have a confluence of issues, none of which were caused by accidents, stupidity, or anything other than dumb luck. And if you dare to claim it’s my own fault or attempt to blame me, no police officer in the world would arrest me for smacking that attitude out of you. Not one. What I have – quite literally, you have better odds of winning the Ohio Classic Lotto. (1 in 13,983,816 for reference.)

But the real problem is the people who do this, and then advertise that you are laughing about it and actively mocking people when this is quite literally a life and death matter for them? To laugh at others for taking it personally when you are talking about things that affect them directly and personally? That is the height of everything that is wrong with this country, and everything that is wrong with you. Every last person who has ever laughed at anyone for taking personal offense to having their rights attacked by people using bullshit political “beliefs” that they didn’t even have till they were fed to them in a soundbite.

All of you need to take a long hard look in the mirror. It isn’t funny and never will be. The cracks at their expense are not witty, funny, intelligent, or well informed. How would you like it if we pointed and laughed at you when you were about to fall off the side of a building? Exactly. And most especially you are not in the right by any measure – no matter which side you’re coming from. Because when you do these things? Yeah.

You’re just a dick. 

I don’t like rewriting posts any more than I like retconning in my fiction. But you know what? I screwed up, bigtime, and I admit it. I’m genuinely sorry, because folks got the wrong message as a result, and that’s on me. It was never meant to be a personal attack. I was literally still shaking with rage and frustration when I wrote this a full day later, and, well, I screwed up. I know better than to write angry, and I didn’t check myself before I wrecked myself. (There. Humor taken care of.) That’s the truth of it.

But I do stand by what I was trying to say, and failed at previously. This isn’t an issue of one person saying one hurtful thing. This is a pattern of conduct by people, especially on Twitter. I have to actually unfollow some people for a few hours (not even who you would guess) when they start talking politics, because if I don’t, well.. it wouldn’t be pretty. The ignorant things you think are somehow okay to say about things you’ve never actually dealt with because you self-justified with quasi-religious political beliefs and learned “all about it” from talking heads on TV or radio? Are absolutely not okay to say. They do hurt people, and they are offensive to people.

I’ve been standing by silently on the sidelines for a long time. And you know, I’m tired of people saying that I and people like me are second class citizens, and of being made to feel like a doormat. I am already being stabbed, beaten, dumped on, and shit on enough as is for even flimsier crap – on a level people do not and should not comprehend. By all rights, I shouldn’t be alive today. Just because I don’t advertise what’s going on, doesn’t mean things aren’t going on.

The Setup in Phil’s Network

For those of you unfamiliar with my setup at home, it’s a mix of “very typical for a home user” and “holy crap that’s complicated.” I have two Synology systems; a DS410 and a new DS1511+. These provide storage over CIFS/SMB2, NFS, and iSCSI to Windows 7 Ultimate, Windows 7 Pro, VMware ESXi, FreeBSD 8, FreeBSD 9, and Linux. That’s the complicated part. I also do not have LACP available, so my DS1511+ is only using both interfaces independently and doing multipathing.

The simple part is having various file sets available on shared folders; primarily my substantial music library, and the large number of documents and files for the startup. There’s nothing all that fancy going on there. Or at least there wasn’t. Then Synology was kind enough to send me a DS1511+ equipped with 5 1TB Seagate disks. So I went “hey. These things are real close to an Enterprise feature set. And they have this new CloudStation stuff in DSM 4.0! Why am I not using this to my advantage?”

So, let’s talk about doing exactly that. And why I love Synology even more than before.

Seriously Good Hardware

When you unbox the DS1511+ you’re going to notice something immediately – this box is heavy. Not just reassuringly heavy, either. You feel like you’re handling a mini tank, thanks to a solid steel disk frame, a heavy gauge steel cover, and a very sturdy plastic fascia. Unlike some other NAS options, the DS1511+ doesn’t feel cheap or even mid-range. It looks and feels downright expensive. You feel like you’re unboxing a new high end workstation instead of a surprisingly affordable NAS. (You can get the DS1511+ for just $770 from Amazon.)

The disk trays are hot-swappable and use a double-lock mechanism to prevent accidental removal. There’s a latching tab plus a push-lock, all made of heavy duty plastic. The trays themselves are thick plastic, but with just enough flex to make sure they don’t crack when you handle them. This is a box you will want to work on, just to marvel at the job the hardware engineers did. It’s like they took even the minor quirks with every preceding model, and fixed them aggressively. I’m not a huge fan of the DS410’s trays – I absolutely love the DS1511+’s.

CPU horsepower is leaps and bounds above my DS410; there’s a dual core CPU clocked at 1.8GHz in here. If that’s not enough to get you excited, how about upgradable memory? When you remove the cover, it’s right there on the system board, a single SO-DIMM socket that will happily take up to another 2GB of memory for a total of 3GB. But my favorite touch here is the power cable socket. It shows you just how much attention to detail Synology put into this hardware. It’s deeply recessed with a stepped receptacle for a standard power cable. You have to really shove it in there. And it stays in there, thanks to that stepped receptacle making it very difficult to accidentally wiggle or yank the cable back out. That’s right – the power cable retention is built right into the socket itself, with no annoying clip.

There’s another detail of the DS1511+ you’re almost definitely going to overlook; the two indentations on the top of the cover. That’s function, not form. You see, those indentations are spaced exactly to hold the feet of a Synology DX510 expansion unit. So why are the feet on the DS1511+ spaced to stack as well? Because with two eSATA expansion ports, your DS1511+ sits between two DX510 expansion units giving you a total of 15 disks. Sure, you can put the DS1511+ on the top or the bottom too. I just think the middle position looks best.

By the way, the DS1511+ likes the number two. Two eSATA expansion ports. Two pairs of USB ports. Two Gigabit Ethernet ports. Two 80mm low noise fans. But it only has one DB15 port – the photos show it exposed, but it will ship covered. You can possibly get it off, but personally I recommend just leaving it be. I didn’t even try to figure out what it did, which is probably nothing at all.

In other words, the hardware is going to make you a very happy owner, before you’ve even plugged the system in. This is a box you are going to be proud to display in your office.

Double Thin iSCSI is In!

Rule number one of Thin Provisioning in the Enterprise space is simple – don’t double down thin provisioning. As in, don’t use thin provisioned VMDKs and Guests on a Thin Provisioned LUN. Well, with Synology, throw this out the window. Sort of.

Let’s start with why I’m using and able to use double thin provisioning to get even more capacity out of my Synology DS1511+. First, I’m using iSCSI LUNs as files on an existing RAID5 volume. Second, I don’t have a need for high write performance. Third, most of my VMs are pretty small when thin provisioned – under 10GB all said and done. This makes thin provisioned iSCSI LUNs and thin provisioned VMDKs a workable combination.

The catch here comes in with parts two and three. The write performance of thinly provisioned iSCSI LUNs is low; it can dip down to 10MB/s, while read hits and sustains over 100MB/s. On a single Gigabit interface from 7200RPM disks. If that didn’t floor you, well, I don’t know what will, since that’s pretty much the effective limit of GigE. As the underlying volume gets toward full, write performance may drop further still.

What this means for you is this – for your home lab where most of your disk operations are read? The performance hit on writes is well worth it. It really is. If you have to have thick provisioned VMDKs, well, use thin provisioned iSCSI anyway! Again; once written, read performance is easily at or just past 95MB/s per interface. Your only penalty comes in when you’re doing a lot of very long writes.

Just How Awesome is the iSCSI?

Despite being able to go on and on, I truly lack the words to describe just how fantastic a job Synology has done with the iSCSI implementation on the DS1511+, especially in DSM 4.0. Beta it says, Production it behaves. I decided to feign going in completely blind on both sides; in other words, act like I didn’t know anything about iSCSI.

Setting up the LUN on the DS1511+ wasn’t just trivial. It was almost too easy. Go to Storage Manger. Create an iSCSI target – I named it for my ESXi system, and replaced the IQN with the hostname of the client. Checked “Enable CHAP” and gave it a name and password. Then I created a new iSCSI LUN as regular files, named it, left Thin Provisioning as yes, and changed the capacity to 100GB, and clicked finish. It’s that simple on the Synology side. I now had a secured iSCSI target and associated LUN.

From the VMware side, well, it was even easier. Enable iSCSI on the appropriate LAN interface. Edit properties of the iSCSI HBA. Go to Dynamic Discovery and add the Synology as an iSCSI server by hostname, configure the CHAP settings to put in the username and password, click OK twice, click close, go back to Storage configuration and rescan. Format discovered LUN and we’re done.

If you want to truly amaze yourself, do it on Windows 7. Same procedure up to the part where we switch to VMware. On Windows 7, you have to go to Administrative Tools to find the iSCSI Initiator. Then all you have to do is click ‘Refresh’ on discovered targets, select the correct target from the Synology and click Connect. If you set CHAP, that’s under Advanced – Enable CHAP log on, enter user and password, done. You now have an iSCSI disk on Windows – the one catch being that you have to go to Disk Management to format it. Right click on Computer, go to Manage, then Disk Management under Storage. Format it, assign a drive letter, and done.

No, seriously. It really is that simple. Even if you have absolutely no experience with or idea how iSCSI works at all, you can get this up and running in less than 10 minutes. Create your new guest on the iSCSI datastore and well, that’s that. You’re running on iSCSI. The Synology doesn’t suffer from false-thin either, so when you format NTFS or VMFS, it stays nice and thin.

DSM 4.0 is Canned Awesome. Here’s why.

I could go on for days and days and days about all the other fun things you can do with your Synology. LDAP server? Yup. OpenERP running on your DS1511+? I tested it out, and not only did it work great, it left ample CPU and memory available at the same time. WordPress? There’s a package for that too! But you already have these things on DSM 3.2. So what’s different in DSM 4.0?

Let’s start with key modification number one: breaking out what used to be preinstalled unremoveable packages. The first time you run DSM 4.0 you’re going to immediately go “wait, where’s DownloadStation? Where’s Media Server? What the hell happened here? I lost stuff!” No. Synology realized that not everyone uses every package, and preinstalling all of them didn’t always make sense. Go to the upper left icon, click Package Center, then go to Available – hey, there they are! I wasn’t expecting this change at all, got caught completely by surprise, and fixed it without searching forums. It’s perfectly logical, and makes a tremendous amount of sense from a development perspective.

See, when you divorce packages like Download Station, Cloud Station, and so on from the core operating system you do two big things. First, you make it easier to debug and fix problems – fixing bug in Download Station means updating the package, instead of having to update the entire operating system. Second, you make it easier to develop everything going forward. You can add all kinds of neat things to packages and quickly implement them, without having to do an entire OS build around them. That means an update cycle that can be cut to a few weeks, instead of the months of testing required for an operating system build.

When it comes to DSM 4.0, there are changes under the hood, but I don’t want to get into them. Why? Because for DS1511+ owners, they’re not going to see too much here. Performance improvements are negligible in my experience, simply because DSM 3.2 is that awesome. Cache performance seems slightly improved, but it just doesn’t matter – it’s already going to be capable of more or less running at 90MB/s+ per interface consistent on 3.2 or 4.0. There’s no new RAID types – personally I like RAID5, but you still have all the options of RAID0, RAID1, RAID10, and RAID5 with Hot Spare and the goodies like Synology Hybrid RAID (SHR) and RAID6. (Suddenly the DS1812+ with its 8 disks makes even more sense, doesn’t it? 6+P+Q is quite delicious I hear.)

I wanted to write about Cloud Station, Synology’s new cloud service that quite literally turns any Synology running DSM 4.0 into your personal cloud server, but I couldn’t get it to work. The reason being that I couldn’t get my firewall to behave quite right, so it’s entirely on me. The writing I have read on it, is that it is fantastic. Part of the magic is that the Cloud Station creates and updates a Dynamic DNS entry provided entirely for free by Synology. Yes, FREE. The whole service package is FREE as in COSTS NO MONEY as in NOT AN EXTRA COST FEATURE and you get the idea. Go to Control Panel, go to ezCloud, Enable DDNS support, select Synology as your service provider, register, done! There’s some iOS goodies available, but obviously, I’m not an iOS user.

The other free cloud-like service is Synology’s new zero cost (yes, ALSO free) monitoring. It’s quite simple. Create your username and password. Register your storage system with Synology by serial number. Go to the new MyDS Center and login. For DSM 4.0, you need to configure ezCloud with the correct login information, but once you have? Your Synology’s health is now remotely monitored with callhome capability – how cool is that? Very, very cool. Speaking of…

What happens when it breaks?

If you follow me on Twitter, you know that my Synology DS1511+ broke rather badly, rather early on. I’m not going to name names – SEAGATE – but somebody, and I’m not saying who – SEAGATE – shipped a defective disk. This disk from someone – NAMED SEAGATE – began remapping sectors silently and writing uncorrected CRC errors from day one, and then experienced mechanical failure due to a platter defect. This in turn broke my DS1511+.

Now to be 100% clear, NONE of the problem was Synology’s fault, nor was it any issue with DSM. The disk failed tests, but wasn’t doing so properly. (Yes, it was failing failure mode.) This resulted in silent and undetectable corruption all over the DS1511+, which in turn, basically caused total data loss. Not because of DSM or the Synology but because the drive wrote bad blocks to itself. That then proceeded to contaminate the rest of the array and the base operating system, because the drive was reporting these bad blocks as OK. Thanks for that, Seagate.

So first and foremost, Synology support may be slow compared to some others – they usually took about two days to get back to me – but they’re also not people reading from a script or giving you canned answers. They do their homework, they test, they reproduce whenever they can. They actually know the product they’re working with, and they take the time to make sure they get the right answer instead of the easy answer. They could have thrown up their hands and said “reset the unit to factory and reformat” at the first sign of OS level trouble, but they didn’t. We spent well over a week working on the problem before deciding the best course of action was exactly that – reinstall the base OS and format the disks. And when we did that, it wasn’t because it was the shortest path. It was because it was the only way to resolve the problem.

So all my problems came down to one very, very bad disk. The chances of you running into this problem yourself are approximately between nil and zero. (Or should be.) You’d have to have a disk installed that should have failed manufacturer QA, then run full exercise tests with that disk including full writes and have it pass, then another dozen steps before mechanical failure to maybe have a chance of reproducing.

As to Synology’s support, well, it’s a love/dislike thing – I can’t say I hate it. I only hate it when support drops the ball or ignores the problem. (Even when there’s a significant delay due to problems RMAing the faulty disk.) Synology has very knowledgeable staff in their support department, they’re dedicated, and their primary concern is getting the problem resolved – these are the things I love. They also don’t drop the ball, and they have never once replied with just a canned answer, or failed to understand my concern or question.  What I dislike is the speed and system, unfortunately. There’s no good way to track the status of your open ticket, and as I said, support can be a while getting back to you. I know that Synology is working on the speed problem, and I’m hopeful that with the new MyDS center the system will improve quickly.

As for fixing a bad disk in the DS1511+? It’s near enterprise style failure handling, no joke. When your disk fails (well at least when it fails properly) the Alert LED lights amber, you get push notifications and emails, and a pretty mild buzzer goes off. Sorry guys; I’m used to the 105dBA of an LSI Elite1650. (Now THAT’S a RAID alarm.) It’ll grab your attention in anything short of a data center though. Failure thus identified, you don’t even need to go into the DSM management interface. Identify failed disk. Push up the latch lock, push in the latch to unlock it, and simply remove the failed disk. Swap the new disk into the tray, insert the new disk, and it automatically starts rebuilding the array. For a 5 x 1TB RAID5, it’s very quick – it took less than 6 hours to perform a rebuild when I tested it.

Since replacing that particularly obnoxious faulty disk? I have had exactly zero problems with my DS1511+. Operation has been flawless, without so much as a single hiccup on any protocol or for any reason. And believe me, I have tried some things that by all rights, should have at least made it grind to a halt. Nothing doing. (And of course, my DS410 continues to be the bastion of reliability, while everything else in my office breaks. Go figure!)

I heard something about multipathing?

You might have. Were you followed? No? Are you sure you weren’t?

Remember how I said I didn’t have LACP? Well, I still don’t. But the DS1511+ has two Ethernet ports, and I have two free ports on my switch, so may as well keep things even right? And while we’re at it, let’s do some interesting shenanigans just to see if they can be done. Turns out, they all can be.

Step one, two interfaces connected, two different IPs. Step two, setup internal DNS with individual A records plus an A record that covers both interfaces. (In my case it’s lan1.ds1511, lan2.ds1511 and ds1511 for the combined.) So what happens when one does this instead of LACP? Well, hosts hitting the “ds1511″ A record will hit either interface, depending what record DNS returns first. It’s a primitive sort of load balancing. But what else happens?

Well, iSCSI turns from “this is pretty cool” into “wait, I can do WHAT with the DS1511+?” Perhaps you have heard of this thing called MPIO. Yes, if you configure the DS1511+ on two separate switches with two separate interfaces, your performance is generally going to be capped at around 90MB/s. However, you now have an actual MPIO configuration. No, seriously. When you look at it in VMware, you’ve got two paths to the iSCSI LUN. I haven’t quite figured out how to get Active/Active Round Robin reads working right though – but that’s a matter of VMware tuning and loading it properly. (And probably upgrading to ESXi 5 too.)

More to the point, this also works on Windows. Unless you have multiple Ethernet interfaces on either client, you’re still going to see around a 95MB/s hard limit – especially with 6GB+ files. Unfortunately, for some reason, despite having two GigE interfaces on my desktop and two paths to the DS1511+, I couldn’t get Windows to not be frustrating about it and refuse to connect to more than one interface. However, when I forced the issue by creating two separate connections explicitly? Synology claims a maximum read of 197.8MB/s – well, they have a better switch and cables than I do. I only hit a combined read of 184.1MB/s. Trust me though – with a good switch and better cables and client interfaces (curse you, Realtek!) you will hit 197.8MB/s easily.

So what’s the conclusion other than “Phil loves it”?

Let’s talk about use cases. Synology pitches the DS1511+ as a great fit for small businesses. And it definitely is. Especially with features like backing up to Amazon S3 and HiDrive, backing up between Synology units (like say, a DS1511+ on DSM 4.0 backing up to a DS410 on DSM 3.2. It works!), incredibly easy but powerful management, and the list goes on.

Home office? Yes, without a doubt, especially with EzCloud and the backup features. Your massive media collection? The new Media Server is fantastic and the indexing works even better on DSM 4.0. Home lab? Absolutely recommended for VMware and Citrix – certified compatible, behaves nicely with clusters, and positively screams. And for every scenario, it’s the only 5 bay NAS that has any scalability.

No, really. I did some homework on the competition in 5 bay high-end NAS. To compete you had to have two GigE interfaces and support CIFS, NFS, iSCSI and have officially supported applications comaprable to the defaults in DSM 3.2 (so AD integration, DownloadStation, Media Server, and at least one other.) You can guess which shiny magic black box utterly failed to make the cut.

Out of the three remaining serious contenders, one had to be evaluated based on their 6 drive model – which offers no external expansion capability whatsoever. None. Attaching a single USB drive, does not count. One offers only one eSATA port and doesn’t support or even permit attaching anything to it – then “stacks” by requiring you to buy more base units that do software RAID over iSCSI. Er, say what? So why did you even include an eSATA port in the first place if you won’t permit anything attached to it?

The remaining competition offers two eSATA ports, just like the DS1511+. They even have a list of supported devices you can attach to these eSATA ports. They even make some of these devices. They more or less claim to be the equal of the DS1511+. Until you realize that those eSATA ports only support one drive devices, period. That’s it. You can connect a total of two, specifically supported USB+eSATA device that can only contain one single disk.

That leaves the Synology DS1511+ as a tiny god; it is, in fact, the only 5-bay NAS where the manufacturer not only intended real eSATA expansion, but also makes and supports that expansion. The maximum of the competition is a total of 2 additional drives. Not devices; drives. That’s not scaling – that’s wasting hardware.

The DS1511+ can support a total of 15 SATA drives (with 2 DX510 expansion units) and 4 USB 2.0 devices for a total of a staggering 19 disks. You can push it past 20 if you use USB devices that RAID0/RAID1 two drives – pushing the DS1511+ to three times the capacity of the competition. It also makes it the only 5 bay NAS on the market with any scalability at all.

Forget “not to be taken lightly.” When the DS1511+ walks down the street, every other 5 and 6 bay NAS dives for cover, praying their owners don’t notice it. Maybe they can match the performance. Maybe they can match some of the features. We wouldn’t be here if Synology settled for that. When you consider the price – under $800; the performance – best of the best; the features – more than anyone else; the compatibility – every OS under the sun including VMware and Citrix; and the expansion? The DS1511+ isn’t just in a class of its own – they had to build a new school just for it.

So why are you still here? Go buy one. Better still, go buy two and build a really awesome home lab – or just some DX510’s. No qualifications, no caveats, no “unless you do this” or “except when that.” I wouldn’t hesitate to recommend the DS1511+ to anyone looking for the best possible NAS for home or office or small business, and I wouldn’t recommend anything else at all to anyone who wants an expandable NAS. (Except another Synology, say, the new DS1812+ or maybe a DS3612xs for those who need dual 10GbE.)

And if you buy one, pretty please use my Amazon affiliate link above. Or just go direct to Synology – just tell them @RootWyrm sent you.

And the Disclaimer!

Synology was kind enough to send me this DS1511+ (including disks) at no charge. All they asked in return was I write about my experiences with it, positive and/or negative. If you follow me on Twitter, you know just how vocally frustrated I got with the problems stemming from that bad disk. I hit a rough spot, and I won’t lie – it was incredibly frustrating. But as ever, I evaluate based on the whole and don’t let “hey this is free” interfere with technical facts. Just gives me an opportunity to evaluate support! And since fixing that disk, the DS1511+ has continued to impress at every turn, and refused to even admit it had a problem.