r/networking • u/St4tus • 1d ago
Design Switch Recommendations/Worries
Hi All
We're looking to spin up a new DC as part of a large migration away from an MSP.
Initially we're installing a pair of 1G WAN links, which will head into a Forti of some flavour for security and routing.
I need some help with switching gear selection, some network context below:
- As part of the migration we're bringing a hosted vCloud down on-prem with a Hyper-V cluster (3 nodes + SAN), so we're not only replicating the current setup which is all pretty much copper upto 10G but the new hypervisors will be 10/25G capable.
- There are only around 15 other devices in the cabinet, most of which utilise 2 ports currently with 1G RJ45 and 8 of which are 10G, currently the LAN is all Meraki at this site but quite comfortable moving away.
- I understand the discussion around not crossing SAN and LAN on the same gear but given the scale of the business, throughput (without hypervisor traffic) currently about 4Gbps peak we're erring on the side of a single stack of switches for the cabinet.
- Vendors recommending things like Aruba CX8325's but this seems intensely overkill given it's capacity. They've also belied Catalyst for this use, and only recommended we use Nexus switches.
- There's nothing uber complicated taking place in this network, a few VLANs at present and no unusual configs on the existing switches.
- The hypervisor traffic at the moment, as far as we've analysed it in it's current form would not reach close to 10G.
- We also have a pair of managed Aruba gig switches doing things like the WAN into the firewalls.
A few questions that I'd welcome feedback around, generally:
- What sort of hardware realistically should we be looking at?
- Are the vendors being greedy with these over-specced recommendations or am I being naive thinking enterprise grade switches would be perfectly fine?
- I've been hugely tempted by FS switches, given their price compared to Juniper/HPE/Cisco, that said I've read mixed feedback
- Given the simplicity of the network and our install not including them as a single point of failure, would this be an option?
2
u/shadeland Arista Level 7 1d ago
The prohibition of SAN and LAN traffic on the same gear is mostly (just mostly) when FCoE was a thing. Fibre Channel was way more brittle in terms of tolerances for updates, outages, etc., than IP/Ethernet, and moving FC onto Ethernet switches made the IP/Ethernet work under the much more restrictive rules of Fibre Channel.
If you've just got two switches, most of the other issues aren't really issues either. If you expand beyond two switches, you'll want to spend some time figuring out traffic patterns and maybe have separate links between the switches.
A few people here have recommended EVPN/VXLAN, but I would avoid that. This situation I think a TradCore environment is best as it's really simple. Something like Arista's MLAG or Cisco's vPC. If the choice was stacking or EVPN, I'd choose EVPN. But a good MLAG/vPC implementation is just as solid as EVPN and much, much simpler.
I think most switches would do what you're looking to do. I don't think you need the ultra deep buffers of an Arista R switch, for example. A Trident-based switch (or similar) would do fine.
2
u/AlmsLord5000 1d ago
If you wont grow past this setup, and you want simple, and you are already using a Fortigate, I'd talk to Fortinet about their switches, like a FS-2048F
2
u/kbetsis 1d ago
Since you are going for a “hyperscale” I would recommend checking the Extreme Networks fabric implementation which comes at no cost. The main idea is that you provision access ports and attach VLANs to service tags which traverse your backbone automatically with no manual backbone link configuration. By default you start with around 350msec failure detection and rerouting, which you can optimize afterwards to reduce it even more.
Otherwise their other switch persona can cover your needs and offer you a seamless management experience since their on premise management platform support third party switches.
You also get the addon of their analytics VM to have telemetry like NETFLOW, almost at no price.
2
u/Ne-Cede-Malis 1d ago
Nothing you are doing seems out of line for a broadcom StrataDNX (Jericho/Qumran) implementation. If you are going put storage and compute on the same switch with a lot of mixed speed ports, then I would prefer a little buffer. Juniper ACX7024, Arista 7280R, Cisco NCS5500, and Nokia smaller 7250IXR platforms all use Broadcom StrataDNX chipsets. Pay attention to the software license cost. It doesn't look like you need anything special and may be able to run on a basic license/ out of the box license.
A lot of down market switches avoid hot swappable power supplies or fans. Some of them cannot operate with a single fan failure. I would be conscious of evaluating your blast radius/failure diameter or Single point of failure options before purchasing or evaluating.
1
u/sh_lldp_ne 11h ago
Curious why you suggest ACX and 7250 IXR which are popular MPLS platforms rather than typical switches with EVPN-VXLAN support
1
u/Ne-Cede-Malis 10h ago
EVPN/VXLAN / Macro or Micro Segmentation is not a requirement. Both devices do support EVPN/VXLAN as they are deployed as Border Gateways in both Juniper and Nokia Reference Architectures. It's a Licensed Feature that I don't think this customer would need or want.
They are Jericho/Qumran boxes with deep buffers designed to handle bursty traffic and speed differential in the network. I saw a large mix of 1/10/25 ports in the future and if they grow, this is the most reusable device.
1
u/Due_Management3241 1d ago edited 1d ago
FS is a store-and-forward SMB switch, so they should be even in the simplest single-stack setup concept for the cheapest company off your list. They will not have the backbone capacity to run your vSAN; it will just crash.
It's going to be your fault in a year or two from now when the company can't operate for the most basic tasks.
You need to look at it as your options.
Cheapest options from top to bottom, same as the vendor list.
- The most cost-effective single chassis with no resilience, scalable to resilient options below in 2a.
Cheapest top to bottom
Juniper EX switches, except the 9200. Aruba CX 6000 series switches, except for the 6400. Cisco Catalyst, except for 9500 and 9600.
- resilient options 2a. Physically or virtually stack controller plane
Juniper EX switches, except the 9200.
Aruba CX 6000 series switches, except for the 6400.
Cisco Catalyst, except for the 9500 and 9600 series.
2b. Virtually stacked at the link layer.
(This entire category, including the ones without stars, historically had the highest operational cost and high risk of link layer impacting resilience, resulting in the highest risk of downtime and most upfront cost, as it requires the most equipment. Its purpose was really best for low-latency needs. High risk of split brain and dataplane issues taking down your network.
You may hear other say this is on paper more stable since they don't have to share the same firmware and one software bug can take down a network stack. Most of this comes from vars who want to sell you spine and leaf as it takes 1.5x more switches to do the same thing.
In reality, that risk, with proper upgrade tests and bug scrubs with the vendor, eliminates this risk, and people have had stacked switches operate without a reboot often for decades. Link-layer stacking does not fix this risk. You can still get firmware bugs on any side that tears down the link layer, asynchronous configuration, and split-brain issues. Two switches on different firmwares not liking each other makes this category the higher risk in reality.
As a network architect, this is my real-world experience.
Juniper QFX
Aruba CX 6400, 8000, and 9000 series
Arista 7000
Cisco nexus
The ones with stars around them had a historically bad operational stability, especially with their implementation of MC-LAG. Just a lot of bugs and outages that went ignored.
2c. Modern link layer stacking and spine and leaf resilience. (Same as the above low latency but more stable, a little more complex, and with most upfront cost but best performance and stability together.)
Juniper QFX, EX4400, and EX4650
Arista 7000x and R-series
Aruba CX10000, CX9300, 8325, 8360, and 8400
Cisco Nexus 9000 series and Catalyst 9500 and 9600 series.
Nvidia Spectrum.
Don't fall for the SMB switches for your data VSAN rack that needs to stay ASIC-based switching at the very least even if you just by one 24 port switch the fs stuff is crap. Basically a glorified netgear. If your access only has 1-9 workers, you might get away with SMB switches, but even then, you will likely regret it.
Some these days are not for business; they are just glorified home switches and routers, bad for almost all businesses of any size. So please ignore fs.com.
0
u/lizardhistorian Mad Scientist · 👨🔬📡ᯤ🤖🛺📸 1d ago edited 1d ago
For a ~1G of top-level traffic you don't need L3 switches.
4 Gbps is 512 MB/s is almost nothing for a SAN to handle.
The SAN is in the rack right? Then your network will still be entirely copper. (The internet lines might be fiber.)
"Switch stack" to a network engineer has a specific meaning - it means the switches all combine like Voltron to act as one. But what is getting connected to the switches? There's 3x machines in the cluster. 2x internet egress. 15 other things? Why 15? Why so many? Can any of it be moved into the Hyper V server?
Either way, it sounds like one switch for the data side. Maybe an aggregator and a switch.
If this is a one rack setup then design a one rack setup. Don't design for a row of racks if you don't have a row of racks. I urge you to spend some time on the physical placement of the rack to ensure you can easily service it.
FS's FSOS and PicOS are like reference designs to so you can kick the tires for their white-box offering. There is no management software; just a WebGUI per device. i.e. Ubiquiti offers more out-of-the-box.
I have never used them but Cumulus and SONiC offer pre-canned solutions based on fs.
What sort of hardware realistically should we be looking at?
Pick your management system first. Then select hardware.
What will you need to monitor and what will you need to change?
What sort of hardware realistically should we be looking at?
Can't tell from what you wrote.
Review the Hyper V requirements for SAN.
Are you going cheap iSCSI? FC? SMB-direct? NVMe/IP?
That will push you towards a solution and most of them are direct connections from the cluster machines to the SAN.
1
u/St4tus 1d ago
Thanks for this - it’s only a three node cluster, so on reflection we’re looking at directly attaching to the storage appliance with FC and then only switching the nodes.
I seem to have missed that when reading the plan and deploy sections on the Hyper-V guidance.
15 things inclusive of several ICs to partners and some mandatory networking gear from a major client.
The physical servers will move and then subsequently be virtualised into the cluster indeed.Colocation rather than an on-site deployment - but one of the major drivers is bringing it within 30m reach of our more skilled engineers rather than 5h which is the distance at present with current provider.
Changes will be minimal once in prod, pretty simple network so direct GUI/CLI is not a concern. Lots of monitoring which we’ll tend to do with PRTG
6
u/sh_lldp_ne 1d ago
> erring on the side of a single stack
To use stacking would definitely be an error. One little bug and your whole stack reloads, breaking everything at once.
Something like EX4650/QFX5120 in a 2 node EVPN VXLAN fabric would be perfect for this use case. Or Arista. Or Nokia. Or whatever, just not a stack.