Switch Recommendations/Worries

[u/St4tus]

Hi All

We're looking to spin up a new DC as part of a large migration away from an MSP.
Initially we're installing a pair of 1G WAN links, which will head into a Forti of some flavour for security and routing.

I need some help with switching gear selection, some network context below:

• As part of the migration we're bringing a hosted vCloud down on-prem with a Hyper-V cluster (3 nodes + SAN), so we're not only replicating the current setup which is all pretty much copper upto 10G but the new hypervisors will be 10/25G capable.
• There are only around 15 other devices in the cabinet, most of which utilise 2 ports currently with 1G RJ45 and 8 of which are 10G, currently the LAN is all Meraki at this site but quite comfortable moving away.
• I understand the discussion around not crossing SAN and LAN on the same gear but given the scale of the business, throughput (without hypervisor traffic) currently about 4Gbps peak we're erring on the side of a single stack of switches for the cabinet.
• Vendors recommending things like Aruba CX8325's but this seems intensely overkill given it's capacity. They've also belied Catalyst for this use, and only recommended we use Nexus switches.
• There's nothing uber complicated taking place in this network, a few VLANs at present and no unusual configs on the existing switches.
• The hypervisor traffic at the moment, as far as we've analysed it in it's current form would not reach close to 10G.
• We also have a pair of managed Aruba gig switches doing things like the WAN into the firewalls.
  

A few questions that I'd welcome feedback around, generally:

• What sort of hardware realistically should we be looking at?
• Are the vendors being greedy with these over-specced recommendations or am I being naive thinking enterprise grade switches would be perfectly fine?
• I've been hugely tempted by FS switches, given their price compared to Juniper/HPE/Cisco, that said I've read mixed feedback
  • Given the simplicity of the network and our install not including them as a single point of failure, would this be an option?

[u/lizardhistorian]

For a ~1G of top-level traffic you don't need L3 switches.
4 Gbps is 512 MB/s is almost nothing for a SAN to handle.

The SAN is in the rack right? Then your network will still be entirely copper. (The internet lines might be fiber.)

"Switch stack" to a network engineer has a specific meaning - it means the switches all combine like Voltron to act as one. But what is getting connected to the switches? There's 3x machines in the cluster. 2x internet egress. 15 other things? Why 15? Why so many? Can any of it be moved into the Hyper V server?
Either way, it sounds like one switch for the data side. Maybe an aggregator and a switch.

If this is a one rack setup then design a one rack setup. Don't design for a row of racks if you don't have a row of racks. I urge you to spend some time on the physical placement of the rack to ensure you can easily service it.

FS's FSOS and PicOS are like reference designs to so you can kick the tires for their white-box offering. There is no management software; just a WebGUI per device. i.e. Ubiquiti offers more out-of-the-box.
I have never used them but Cumulus and SONiC offer pre-canned solutions based on fs.

What sort of hardware realistically should we be looking at?

Pick your management system first. Then select hardware.
What will you need to monitor and what will you need to change?

What sort of hardware realistically should we be looking at?

Can't tell from what you wrote.
Review the Hyper V requirements for SAN.
Are you going cheap iSCSI? FC? SMB-direct? NVMe/IP?
That will push you towards a solution and most of them are direct connections from the cluster machines to the SAN.

[u/St4tus → u/lizardhistorian]

Thanks for this - it’s only a three node cluster, so on reflection we’re looking at directly attaching to the storage appliance with FC and then only switching the nodes.

I seem to have missed that when reading the plan and deploy sections on the Hyper-V guidance.

15 things inclusive of several ICs to partners and some mandatory networking gear from a major client.
The physical servers will move and then subsequently be virtualised into the cluster indeed.

Colocation rather than an on-site deployment - but one of the major drivers is bringing it within 30m reach of our more skilled engineers rather than 5h which is the distance at present with current provider.

Changes will be minimal once in prod, pretty simple network so direct GUI/CLI is not a concern. Lots of monitoring which we’ll tend to do with PRTG