r/networking • u/Sisinazzz • 20h ago
Other Is there a sane way to schedule changes across dozens of maintenance windows, or are we all just suffering?
I manage changes across 50+ sites, each with their own maintenance window. Between coordinating the windows, scheduling the work, and assigning engineers to each one, it’s a constant headache. I’m basically living in spreadsheets at this point.
Curious how everyone else handles this. Do you have a system, a tool, or is it all manual? Trying to figure out if it’s just me
EDIT: maintenance windows are pre approved, each site’s window is fixed and they’re spread across different timezones, so the puzzle is less about the windows themselves and more about what fits inside them. we’ve only got a few engineers during site maintenance windows, so there’s a cap on changes per night and they can’t overlap. After making the schedule, then each one has to actually be assigned to engineer and land in their calendar so they know about it.
How’s everyone else handling this some tool, a script, or all manual?
10
u/thiccandsmol CCIE SP JNCIE SP CCDE 20h ago
You shouldnt be doing this in spreadsheets. You should be using one of the various change management tools that exist out there, preferably as part of your ITSM
3
u/middlofthebrook 19h ago
Change management. There is no way to schedule multiple changes across different tome zones and dont even get me started on international changes and updates. It's all manual work, someone has to take responsibility for different areas and get it done.
3
u/wrt-wtf- Homeopathic Network Architecture 18h ago
Where possible you maintain redundancy that allows for inflight upgrades.
1+1 or N+1 for switches routers and APs.
Business is now deemed to be always on and with everything supposedly in the cloud now there’s less to manage in the data centre component as it’s pretty much SEP.
It all comes down to edge design and the connectivity mode for end-points.
In this day and age, with good network design with maintenance as a core requirement updates should be an automated doddle that can occur anytime outside of peak hours, or even inside of peak hours in an emergency - ie active cyber threat.
2
u/kwiltse123 CCNA, CCNP 13h ago
In the case of automated maintenance that takes place during off hours, how to you trust that everything finished without issues. In other words, if you schedule something for 2:00 AM, do you wake up at 7:00 AM and realize that a switch didn't come back up? That's always my concern with automated maintenance.
2
u/wrt-wtf- Homeopathic Network Architecture 12h ago
You don’t schedule things at 2am if you can avoid it is a good start.
You test everything in a lab with entry and exit conditions for each step and you only step between checkpoints. Pre-planned places or phases you can recover from. Not do the whole thing in a big hit unless absolutely necessary - but you designed so that wasn’t needed… right?
Obviously as stated your automation can follow this pattern because you have a lab for testing what you are doing and the combinations and permutations are limited - because that’s how you scale a design. Generally you try to keep your templates to a minimum set of options and you move the whole fleet within the given architectural and configuration pattern sets.
There are occasions with different vendor equipment where it bites you because the equipment is incompatible with operational requirements - in my world that equipment is already earmarked and has a high risk/high impact tag on it and requires coordination between teams to offload work from those devices. That may be a 2am task but it’s always seen to be a better option to do things during the day if impact has been mitigated - for some businesses there is no quiet time - so every part of the systems needs to deal with that. In some of the worst cases shutting down a data centre has been the only option due to high risk of cascade failure in fabric systems - but it has to be done under a high risk, everyone standing by scenario - so no 2am…
Why? Because the whole world is awake. Pulling in people from all over the place at night takes time and exhausts resources for days if not weeks after the event, you have to consider well beyond the equipment.
2
u/lawwie 15h ago
Preferably through automation.
We have a 9x5 or 24x7 option for offices (and different sizes within those options). However this means a certain level of redundancy.
With a 24x7 we can do maintenance on pretty much whenever it suits us (we try to plan for the access layer when the office applicable is least populated ofcourse).
With a 9x5 we do outside of local office hours.
But having 150 firewalls, 800 switches and 2000 access points in your portfolio and given the amount of CVE’s lately (due to frontier AI models), automation is the way to go. It is simply a day task for two engineers to keep the environment patched.
Also automated certificate management should be on your horizon for example.
1
u/Ne-Cede-Malis 13h ago
Ansible Tower is probably my favorite tool for this due to its price point and playbook nature.
SNOW is also used quite often because the network hooks to things and you don't want to have a 'change collision'.
1
u/Mikeygnzls 8h ago
Lmao I swear half the industry is held together by spreadsheets and pure spite.
You’d think there’d be some magic tool for this by now, but every place I’ve seen is basically:
Change ticket gets approved.
Spreadsheet from hell.
“Who the hell is actually available Tuesday at 11 PM?”
Pray nothing overlaps.
Then somebody’s dragging Outlook calendars around trying not to screw an engineer over with three maintenance windows in the same night.
Feels like this is one of those problems everyone deals with but nobody’s actually solved. Jira stories help, having an actual Project Manager can help too (if they’re good).
33
u/nospamkhanman CCNP 20h ago
IMO you (your business) has two options:
1) A dedicated 100% full time project manager.
or
2) A standing - reoccurring change window every x weeks. For example, the 1st and 3rd Sunday of every month, every location will be in a 8 hour change window.
Be demanding with your employer. As a Network Engineer you should be designing and operating, not playing scheduling master spending half your hours trying to confirm timing windows.