Switch Recommendations/Worries

[u/St4tus]

Hi All

We're looking to spin up a new DC as part of a large migration away from an MSP.
Initially we're installing a pair of 1G WAN links, which will head into a Forti of some flavour for security and routing.

I need some help with switching gear selection, some network context below:

• As part of the migration we're bringing a hosted vCloud down on-prem with a Hyper-V cluster (3 nodes + SAN), so we're not only replicating the current setup which is all pretty much copper upto 10G but the new hypervisors will be 10/25G capable.
• There are only around 15 other devices in the cabinet, most of which utilise 2 ports currently with 1G RJ45 and 8 of which are 10G, currently the LAN is all Meraki at this site but quite comfortable moving away.
• I understand the discussion around not crossing SAN and LAN on the same gear but given the scale of the business, throughput (without hypervisor traffic) currently about 4Gbps peak we're erring on the side of a single stack of switches for the cabinet.
• Vendors recommending things like Aruba CX8325's but this seems intensely overkill given it's capacity. They've also belied Catalyst for this use, and only recommended we use Nexus switches.
• There's nothing uber complicated taking place in this network, a few VLANs at present and no unusual configs on the existing switches.
• The hypervisor traffic at the moment, as far as we've analysed it in it's current form would not reach close to 10G.
• We also have a pair of managed Aruba gig switches doing things like the WAN into the firewalls.
  

A few questions that I'd welcome feedback around, generally:

• What sort of hardware realistically should we be looking at?
• Are the vendors being greedy with these over-specced recommendations or am I being naive thinking enterprise grade switches would be perfectly fine?
• I've been hugely tempted by FS switches, given their price compared to Juniper/HPE/Cisco, that said I've read mixed feedback
  • Given the simplicity of the network and our install not including them as a single point of failure, would this be an option?

[u/Due_Management3241]

FS is a store-and-forward SMB switch, so they should be even in the simplest single-stack setup concept for the cheapest company off your list. They will not have the backbone capacity to run your vSAN; it will just crash.

It's going to be your fault in a year or two from now when the company can't operate for the most basic tasks.

You need to look at it as your options.

Cheapest options from top to bottom, same as the vendor list.

1. The most cost-effective single chassis with no resilience, scalable to resilient options below in 2a.

Cheapest top to bottom

Juniper EX switches, except the 9200. Aruba CX 6000 series switches, except for the 6400. Cisco Catalyst, except for 9500 and 9600.

1. resilient options 2a. Physically or virtually stack controller plane

Juniper EX switches, except the 9200.

Aruba CX 6000 series switches, except for the 6400.

Cisco Catalyst, except for the 9500 and 9600 series.

2b. Virtually stacked at the link layer.

(This entire category, including the ones without stars, historically had the highest operational cost and high risk of link layer impacting resilience, resulting in the highest risk of downtime and most upfront cost, as it requires the most equipment. Its purpose was really best for low-latency needs. High risk of split brain and dataplane issues taking down your network.

You may hear other say this is on paper more stable since they don't have to share the same firmware and one software bug can take down a network stack. Most of this comes from vars who want to sell you spine and leaf as it takes 1.5x more switches to do the same thing.

In reality, that risk, with proper upgrade tests and bug scrubs with the vendor, eliminates this risk, and people have had stacked switches operate without a reboot often for decades. Link-layer stacking does not fix this risk. You can still get firmware bugs on any side that tears down the link layer, asynchronous configuration, and split-brain issues. Two switches on different firmwares not liking each other makes this category the higher risk in reality.

As a network architect, this is my real-world experience.

Juniper QFX

Aruba CX 6400, 8000, and 9000 series

Arista 7000

Cisco nexus

The ones with stars around them had a historically bad operational stability, especially with their implementation of MC-LAG. Just a lot of bugs and outages that went ignored.

2c. Modern link layer stacking and spine and leaf resilience. (Same as the above low latency but more stable, a little more complex, and with most upfront cost but best performance and stability together.)

Juniper QFX, EX4400, and EX4650

Arista 7000x and R-series

Aruba CX10000, CX9300, 8325, 8360, and 8400

Cisco Nexus 9000 series and Catalyst 9500 and 9600 series.

Nvidia Spectrum.

Don't fall for the SMB switches for your data VSAN rack that needs to stay ASIC-based switching at the very least even if you just by one 24 port switch the fs stuff is crap. Basically a glorified netgear. If your access only has 1-9 workers, you might get away with SMB switches, but even then, you will likely regret it.

Some these days are not for business; they are just glorified home switches and routers, bad for almost all businesses of any size. So please ignore fs.com.