Executives never make mistakes. Well, (as Gilbert and Sullivan put it), “hardly ever.” As a networking consultant working with a wide variety of organizations, I’ve seen a few technically challenging situations that originated at the C-level. The causes: too much or too little involvement in the network technology.
Vendor marketing and sales always make new features and technology appear to be The Next Big Thing, something you have to have. That’s what they’re supposed to do. Good vendors listen to their customers, especially the executive decision-makers, and try to address the needs they hear. That is, they’re going to be telling you what you want to hear. And if you just give them enough dollars, their shiny new solution will solve all your problems. Agile deployment? No problem, they’ve got it! Lower total cost of ownership? Can do!
What can get lost is a discussion of timing, risk, and maturity of the product. New technologies inherently have risk due to immaturity and bugs. There are also usually negative aspects of a proposed solution that the vendor doesn’t mention. Questions to consider asking during a vendor presentation: “What are they not mentioning? Are there hidden costs?”
The right timing is important concerning the risks and maturity of new technology. Your organization’s staff skills and normal workload are one factor to consider. The organization’s tolerance for, and ability cope with, risk and product immaturity and bugs is another factor.
That’s where executive involvement comes in. Here are some “Do’s and Don’ts” for executives for selecting new technology and managing installations:
- Do use your technical staff and consultants in evaluating product and technology risks, especially the unspoken downsides. Don’t just decide to buy with no hands-on testing; products often demo well but can be buggy or exhibit problems when tested in the real world.
- Don’t dictate technology choices before the drawbacks, risks, and alternatives have been adequately explored.
- Do take actions to get staff moving when they get stuck into a technical rut.
- Don’t check out of the process. Keep asking your technical staff hard questions about the pros, cons, and alternatives. And do keep asking “Is there a better way to do that?” and “Is there a less complex way to do that?”
- Do think holistically: networking, server, application, storage, and security are all coming together. Making a security decision in isolation, for example, may very adversely affect the network in terms of cost or complexity. Solutions should be designed with equal input across traditional stovepipe boundaries.
- Don’t allow your network to become a CCIE lab challenge. This happens occasionally in small to medium shops with a highly technical staffer who may lack common sense, or may not have been told that simplicity is a business requirement.
- Do consider pilot trials for any new technology. Implementing a new technology generally makes bugs, gaps, costs, and complexity far more apparent. Giving network management software a thorough pre-purchase 60- to 90-day trial is similarly informative.
- Be careful about vendor-provided ROI tools. If they save you two techs worth of labor but to make the product work you have to hire a rocket scientist costing three times their comp, is that a win?
Here are some concrete examples where one of the above action items did not take place:
- Active-active datacenter is a hot topic right now. But which versions of it are mature, and do you understand the risks? In particular, a spanning tree event in one datacenter can now take down both datacenters, especially if you don’t take defensive measures. Are the benefits worth that risk? Another consideration: stateful firewalls may be problematic in such scenarios, due to asymmetric flows. So small events that change the traffic path can cause dropped connections, which users will perceive as a less reliable network.
- Software Defined Networking is also hot. It offers big opportunities with some risks to large organizations. I spoke up a while back and said “the emperor has no clothes” in regards to “we all need to learn programming.” The issue is identifying what is right for your organization: a vendor-provided and supported solution, several software packages requiring integration, or in-house API programming projects?
- Expensive and antiquated network management products are brought into some organizations because someone did a heck of a sales job, and there was insufficient follow-up with all the relevant technical staff or with consultants before the deal was finalized. A $100,000 product might actually be more useful to the people who will be using the products than the $1.5 million product sold at the C-level that only covers part of the network or datacenter and requires constant care and feeding by costly consultants.
- Security staff decisions can very adversely impact network and server staff, either making it harder for them to do their job (complexity = increased risk, including security risk), or negating design decisions. The most prevalent example right now is deciding that all traffic to and from servers must go through “the Big Firewall.” Problems with that: (1) the new network switches can move traffic 100+ times faster than that firewall, (2) a physical firewall fails to control virtual machine (VM) to VM traffic, and (3) inserting a big central firewall can significantly complicate network design and troubleshooting. This sort of thing happens because of non-holistic design decisions, e.g. assuming that what security proposes to do only impacts the security team.
- Some executive-dictated network designs end up being overly complicated and have increased risk of failure or human error, sometimes because technology or design choices got cast in concrete too soon by the network director or CIO.
- I see Lync deployments getting done by senior server staff with no knowledge of IP telephony. Even in environments with experienced network staff with those skills. Why are the server admins re-inventing the wheel, instead of leveraging in-house expertise?
Things that can help:
- Set the goals, identify potential technical solutions, then get more data on the alternatives.
- Use lab testing and pilot projects — is the product ready for prime time, can staff deploy and operate it?
- Establish a timeline: if you like the looks of a product or technology, try to predict when it will be mature enough for your organization.
- Have good communications, not only up and down, but also across traditional boundaries.
- Use holistic design thinking across IT: how does this affect the other technology teams?
- Keep it simple: use key functions, and don’t get seduced by bells and whistles.
When we do a design, we strive to understand your business requirements, including level of simplicity or complexity. We also try for balance across technology areas, so that no one area bears a heavy burden for the choice made in another area.
For more information about the business value of technically simple, elegant networks, read our Fact Sheet on infrastructure deployment.
Comments
Comments are welcome, both in agreement or informative disagreement with the above, and especially good questions to ask the NFD9 vendors! Thanks in advance!
Hashtags: #CiscoChampion #netcraftsmen #TechDeployment #C-suite
Twitter: @pjwelcher
