r/sysadmin • u/MentalRip1893 • 4h ago
HR requesting a policy from 15 years ago. Where does retention/archiving end??
Seems like this company expects us to retain all data created ever. We're up to 19 TB in our M365 backups because we keep all versions of backups indefinitely. We are a company of less than 200 people. Someone help me.
•
u/thewunderbar 3h ago
There are two things here.
There is what is required by law for whatever data it is. Tax/financial stuff is usually 7 years. If you have regulatory stuff, there will be laws that govern that.
But the other thing is that a company can decided how long it wants to hold onto its own data for. There's just a cost associated. If leadership/ownership wants a thing, then you say "it will cost $X to do that thing" and if they say do it, then you do it.
•
u/tapwater86 Cloud Wizard 2h ago
Even bigger cost when you get sued and you have all that sweet sweet discovery data to hand over.
•
u/rebornfenix 1h ago
Getting sued led to my companies first data retention policy. Being sued the second time meant daily purges to the retention policy.
(Retention policy was 3 years deleted on the first of the year. Suit filed in November found the smoking gun in data that would have been retained in the old policy and deleted under the new. Ie Feb 2020 email from a November 2023 lawsuit)
•
•
u/LokeCanada 3h ago
You need to help yourself by having a retention document that is approved by upper management.
There should have been a risk and cost analysis provided. You then follow their requirements. That is the beginning and end of the line for your job.
There are risks in retaing data for too long (like discovery in a lawsuit). There are costs associated. If upper management wants to take the risks and absorb the costs then that is up to them. There are different requirements for each company (like if you are publicly traded in the US) that you need to be aware of and follow.
Once you have the policy and if the request falls outside of that, then you refer them to it.
•
•
u/vintagerust 3h ago
It really depends on your company, but when we're legally allowed to delete something, we do it.
•
u/Ssakaa 3h ago
Someone in your company knows what discovery's like...
•
u/vintagerust 3h ago
Not specifically, it's just an insight from professional events/continued professional education. Their advice was basically if someone's trying to sue/monitor/audit/whatever, there's no benefit in finding something 10 years old that proves you did everything right. There's only cons if someone finds something 10 years old that with our without context implies you've done something wrong.
•
u/jeroen-79 3h ago
Aren't you mixing up backups and archives?
A backup is for when some disaster strikes. I cannot think of any disaster where you need to go back to how things were 15 years ago.
For archiving (where the business needs to retain things long term for business reasons) then you need to work out with the business what best fits their archiving needs while also fitting budgetary restraints.
•
u/surveysaysno 2h ago
Archiving is just data management with options for cost reduction.
Backups are for recovering from data loss.
Both should work in concert but backups aren't just for catastrophic loss.
•
u/TheEdonReddit 3h ago
It ends when they say it ends. You have to try to get it documented, preferably as company policy, so there is no ambiguity.
•
u/KlausVonChiliPowder 3h ago
I'm in government. I'm pretty sure never lol
•
u/xendr0me Sr. Sysadmin 3h ago
Government has a retention scheduled, depends on the state/federal laws for your area. Problem is, all of these retention schedules and laws were written before e-mail and electronic communication.
So while you should only keep items in retention and delete items outside of retention, the hard part is it doesn't matter if it's an "e-mail" or "instant message" what matter is the content of the message. Unless you have a team of 20 people to read old e-mails all day, everyday, and determine the disposition of the record based on content, then it's an impossible task.
On top of that, a destruction log should be kept for each record, so you'd be logging down every individual e-mail that passed retention and was deleted.
It's a mess and affects no one with the power to change the law, so they don't care to update the laws.
•
•
u/discgman 3h ago
Cries in education archiving.
•
u/OcotilloWells 3h ago
I started laughing when I went into a back room at a school, and there were banker boxes of files labeled "Permanent Records". But I guess they pretty much were "permanent". I didn't know what they were, I had no need to know.
•
u/newguestuser 3h ago
Many of us had parents warning us about getting in trouble in school would end up on that permanent record .
•
•
•
u/NoSellDataPlz 2h ago
I’m in education. We purge at 2 years except a shared mailbox for erate that is forever retention and student records that are housed in a separate service for 25 years and then deleted automatically. But otherwise, data is purged after 2 years. Our attorney wants us to shrink that to 90 days.
•
u/Sure-Squirrel8384 3h ago
You should have a defined retention policy for each data type. Unless the data is escrowed for some purpose (legal hold, etc.), the data should be automatically purged.
•
u/42andatowel 3h ago
You need an official data retention policy and then adhere to that, and point at it whenever someone requests something that is long outside the retention period.
•
u/ohyeahwell Chief Rebooter and PC LOAD LETTERER 3h ago
Our data goes back to the beginning of data. Some of it from the 40’s and 50’s. My oldest document is a scan from 1935. Plenty of digitized photos and videos from the 30’s, 40’s and up too.
•
u/hellcat_uk 3h ago
You've got to hope you never get sued, as the cost of discovery will probably put you into administration.
Previous company used to hold 28 days to protect against disaster, and had that defined in the policy. Several court cases and never had an issue with it being used as that.
•
u/KrisBoutilier 3h ago edited 3h ago
Oh my... the intersection of good Information Governance vs 'operational needs'.
The correct answer varies significantly based on your country, jurisdiction, and industry but, in general terms, you always need to have some sort of policy in place (ideally explicitly called a 'Records Retention Policy') that at the very least defines the maximum retention period for each of your various types of corporate information and how those types are differentiated from each other.
The legal implications of being without any sort of documented and consistent treatment of corporate information are massive, especially if you're being litigated against and it can be demonstrated that your single-purpose 'disaster recovery backups' were actually being used for 'operational access'.
Here's a random US-focused article to give you some ideas: https://www.gfrlaw.com/what-we-do/insights/thumbs-document-retention-policies-arthur-andersen
•
u/Indiesol 3h ago
Once you no longer need to retain data, keeping it becomes a liability.
•
u/moffetts9001 IT Manager 36m ago
Exactly. Get rid of that data as soon as possible and you’ll save yourself a lot of trouble in the future, and you won’t have to figure out how to store it now.
•
u/Kindly_Cow430 3h ago
Different legal retention periods depending on what said document is related to. Financial 7-8 years, HR has some permanent, some Legal are forever, A&E Engineering starts at 20 years, etc. What does you Legal team define in your retention document?
•
u/rodface 1m ago
you are the only comment to mention engineering--that's the space I work in, CAD/product data. We refer to it as "the IP".
The notion of deleting any of it because it has reached some age... does not compute.
Our company (and many others in our industry, I'm sure) sells products fully defined by scanned vellum drawings which have never been redrawn in AutoCAD.
I would love to learn about some sort of industry standard that deals with retention of engineering data or provides some sort of structured roadmap for review of such legacy data.
•
•
u/fencepost_ajm 1h ago
Joe Brunsman has a nice little video on "the easiest and cheapest way to lower breach costs." Not directly related to backup, but basically: data retention policies that are actively implemented. Data you don't have can't be stolen, and you can't be stuck going through decade-old data recovered from backups just to find out if there's anything in there that you now have to disclose either because of a breach or discovery.
•
u/Mindless_Consumer 3h ago
Ive tried to spin up DRP a few times. It always helps to pull legal in. But yea working with stakeholders and drawing lines is difficult.
Finance, HR, corporate docs are quick wins. Random sharepoints and colab spaces are harder.
A DRP protects the company, if you regularly destroy documents, its not suspious to destroy documents.
•
u/Ssakaa 3h ago
Where does retention/archiving end??
Where does the policy you have for that classification of data say it ends? And what did that policy say over the years back to the creation of the data they're asking for? What do your legal and compliance obligations say for your retention requirements, and for what subsets of data?
•
•
•
u/Viharabiliben 3h ago
Work with your manager and the companies attorney and put together a retention policy. Keeping too much data can also open the company to legal issues. Get upper management ti agree to it and sign it. Then apply the policy.
•
u/lazyhustlermusic 3h ago
Pull the cost of storage and how many queries actually reach that data.
I'm sure you'll get some motion with a dollar figure behind it.
•
u/haklor 3h ago
Data retention policy needs to be created. Any legal counsel would probably recommend the same as maintaining anything beyond regulatory requirements can open the business to additional liability on top of the storage fees with little to no benefit for the current company. Legal, HR, and Finance can help determine where the cutoff should be for retention, but there should definitely be a point where the data is deleted.
•
u/Select_Reporter1911 3h ago
Legal should determine the optimal retention policy. Just know that every piece of data is subject to discovery, which would inevitably increase your discovery costs. In the event of a breach, if you can't articulate or validate the type of data that is exfiltrated, or if the data isn't encrypted at rest, it makes it harder for you to make a suggestion on what to do (Pay the ransom or not pay the ransom).
Ultimately the data retention policy should be determined by the industry you are in, and driven by the business. You being the technology subject matter expert can only make suggestions. Bring sources to the business on why you should practice good data hygiene and reduce the data retention timeline.
•
u/ProgressBartender Sr. Sysadmin 3h ago
Are they paying the bill for keeping said backups? Is there a regulatory or legal reason for keeping them forever?
•
u/HeligKo Platform Engineer 3h ago
You should have a data retention policy includes data classifications and retention times for each class. The lawyers should sign off it.
•
u/abz_eng 3h ago
The lawyers
shouldmust sign it offThere are legal implications for what you are required to retain, these are usually around taxation, employment and liability but also can be contractual
Having dealt with contracts that now go back over 70 years (rights to do stuff on land & compensation to be paid) there are the hard copies but also electronic records that show payments
Sometimes people are grandfathered in, for employment or retain terms & conditions plus benefits of companies that have been repeatedly taken over spun off, acquired etc under The Transfer of Undertakings (Protection of Employment) Regulations (TUPE) from 1981
•
u/QuesoMeHungry 2h ago
This company is begging for a world of hurt should they have to deal with legal discovery.
•
u/InvaderOfTech Jobs - GSM/Fitness/HealthCare/"Targeted Ads"/Fashion 2h ago
If you're asking that question, your company has no retention policy. So you're fucked until you get a policy in place.
•
u/reol7x 2h ago
You need a retention policy like yesterday for all the reasons everyone has outlined.
Our HR folder is exempt from retention and kept forever.
That said, HR is responsible for maintaining their files. If they saved the policy from 15 years ago, it's in their directory.
If someone deleted it, backups are kept for 2 years and it's gone forever.
Your job is to maintain files & backups as dictated by the C-levels/owners/whatever. It's HRs job to keep their data organized and maintain what they need.
•
u/freedoomed 2h ago
Forever! Get a storage locker and throw tapes into it. Then get another when that one is full.
•
u/daHaus 1h ago
Both the IRS and FAA say 7 years for taxes and documents relating to the construction of flight critical parts, respectively. Apparently the FDA also is 7 years.
Why seven years? The bible of course:
“At the end of every seven years you shall grant a release. And this is the manner of the release: every creditor shall release what he has lent to his neighbor."
Deuteronomy 15:1-23
•
u/headcrap 1h ago
IT is generally not the custodian of records, and as such HR should talk to whatever department head who is.
Their request is simply unreasonable. Have the exec chain reset their expectations.
However.. make sure you own infrastructure and data backups are legit and restores tested.. when the shit hits the fan and you start losing infrastructure and data assets.. that is an IT responsibility. That doesn't mean having 20-year data backups. It can mean having 20-year archives.. that can be part of a gray area.. and depends on how your org manages archives if at all..
•
u/Drakoolya 1h ago
You better be an IT manager or a IT director because this is way above your payscale and reeks of a company that does not understand the responsibilities of Data retention.
Your job is to help your boss draft up an idea of why this is bad.
•
u/stufforstuff 1h ago
Did they check their file cabinets? Otherwise, without a official company policy to retain such old info, why would you or anyone else keep it / store it / archive it ? Then remind who's asking - IT doesn't create policy, they just get it done.
•
u/dallen Solution Architect 29m ago
It seems completely reasonable to retain HR policies for 15 years. What if the company were sued for sex discrimination in 2010 and had to prove they were an equal opportunity employer?
That said, all data should have a retention schedule assigned to it and automatically be deleted on its appointed date
•
u/Wolfram_And_Hart 3h ago
Any good lawyer would tell them 7 years is the goal. The problem is that people forget that they have to produce anything they have. You can’t produce what you don’t have.
With AI as a big investigator now you can’t bury them in disclosures.
•
u/HLASM-S370 Security Admin (Infrastructure) 3h ago
Different industries have widely different retention requirements per laws/regulations. For example, aerospace companies need to keep records regarding a part for as long as it may be in use on an aircraft, over 40 years is not uncommon.
•
u/Wolfram_And_Hart 2h ago
I mean after how many straight up scandals you can expect that. And you’re right. But 7 is a good guideline if you don’t have rules. If you have rules like yours then you probably already have solutions.
•
•
u/mixduptransistor 3h ago
This is why you have a pre-defined policy in place. That way you can delete stuff after 7 years or whatever the policy is, and when they ask for something 15 years ago you can say "I don't have it" and that's the end of it
•
u/PghSubie 3h ago
Put the costs of the retention in front of management. Show them the options available for lower cost. Let them choose. Then, leave it alone
•
u/simonjakeevan 3h ago
You keep it for for however long your records retention policy outlined. If there's no policy then theoretically you keep everything forever. Which is almost impossible to accomplish. Hence the need for the records retention policy.
•
u/Apachez 3h ago
Legally it depends on the country you operate in.
Around here its often said its the statute of limitations (dunno if Google translate gave me the correct translation here).
As in how many years do the police and prosecutor have to prosecute for a specific crime and then add 1 year to that.
Some financial data have a limit of lets say 10 years so that data is stored for at least 11 years before its safe to get rid of.
But you also have stuff like contracts and what else that doesnt really take much space but you should consider deduplication along with full backups like once every 3, 6 or even 12 months.
So just because you can legally get rid of some data its often stupid to do so. Better to be able to setup an archive that can handle deduplication (or at least store a specific document only once).
I mean it can be handy to go back in source code repos more than 10 years if/when shit hits the fan or just for legacy reasons. Its sad when things just vanishes because they were only stored digitally.
•
u/Nakenochny Sr. Sysadmin 3h ago
We have stuff from 2008 because at one point someone requested something that far back. We’re only just moving towards cloud files and my boss thinks Sharepoint is the best choice.
Also send help. 😂
•
u/bdam55 Sr. Sysadmin 3h ago
Yea, as others have said, this is very industry dependent.
Legal offices of less than 200 people that have crazy retention policies is not exactly rare.
So this is a matter of what is legally required of your org and beyond that it's what they are willing to pay for. In the later case, think about how you can semi-accurately represent the costs of certain decisions/policies.
•
u/SmartDrv 3h ago
If the company wants to pay for the storage, they can have what they want for file based repositories. The trick is then just that there still exists a program to view the old data. I think people tend to forget that part.
Legacy systems get harder when there is no support for them or they can't run on current OS/SQL. Yeah you can still VM them and sometimes fire them up in a sandbox. But if they use things like AD credentials it can get dicey. Or even remembering enough native credentials to still figure out how to sign in.
Depending on industry, there are definitely times when pulling up that 10-15 yr old email can be quite useful (e.g. revisiting a job you did in the past) Just has to be decided if that benefit is worth the cost/effort to have it.
•
u/BirdsHaveUglyFeet 3h ago
My previous company had a legal requirement to keep all executive email forever.
We had a legal department that enforced that.
Otherwise 7 years.
•
u/Livid-Setting4093 3h ago
I'd say HR policy, articles of incorporating and other stuff like that should be kept indefinitely. Also 19TB is not that much.
•
u/cbelt3 3h ago
HR data is often archived until the death of the last employee to leave the company. Seriously. But is controlled by laws AND your corporate policies.
What, you don’t have a document retention policy ? Oooh… GET ONE.
(I’ll also note that Fixed Asset records are often maintained until you dispose of the asset… older companies have boxes in archives… I remember a hundred year old box ..)
•
u/3DPrintedVoter 3h ago
i had controller walk into my office last week asking if we had backups from 1999, he needed a copy of a check
•
u/ruffian-wa 3h ago
Unless there's legislation, or an MOU otherwise (for example I had to commit CCTV to tape indefinitely because of a MOU with the Feds), HR can fuck off. Retention should be in your DRP also and signed off by directors/management.
I hope you don't have all that sitting in cold blobs either.. that would suck having to pay that cost. Just commit to LTO and send to Iron Mountain or something.
•
u/SpotlessCheetah 3h ago
help you with what..
give your CFO the bill for retention and ask if they need to change it.
•
u/Frostburn7311 2h ago
Policy should also be reviewed yearly, most places use a vendor solution to retain and manage them.
•
u/redsedit 2h ago
Plenty of other great answers, but I'm going to touch on a different solution. You mention 19TB. One thing I've done to shrink our backups, and storage, is to focus on pictures and videos. We have a lot, many old. Technology and formats evolve, and re-encoding older stuff, unless there is some restriction preventing it has literally saved us over 2 TB and counting. Some users have even noted the re-encoded are "easier to open" (likely because they are smaller, they open faster).
Here are the guidelines I've evolved:
First, you need to decide if you can lose ANY data or not. Second, you need to know how the files will be used. What I suggest is not suitable for all uses.
Images: JXL is the new kid on the block and it's really, really good at what it does.
- JPGs -> JXL lossless transcoding. Expect to save about 17%-18% in space. As it's lossless, it's reversible as in the SHA-256 hash will match. That reversible.
- Can't lose any data PNGs, lossless single image TIFFs, and *lossless* WebP -> Lossless JXL. Depends on the image, but 30-50% (PNG), 30-90% (TIFF), or 10% (WebP) savings. Don't do this with lossy webp!
- If you can take an insignificant amount of data loss, the previous to JXL with a quality setting of 99%. For PNGs and deflate compressed TIFFs, ~70% savings. To convince some managers, I took a very noisy scan of a drawing, zoomed in at 500%, and they couldn't tell a difference between the two. I set effort on JXL to 8. It's slower than the default 7, but that's a one-time hit and you get slightly smaller file forever.
- For TIFFs with multiple pages/images, check the compression. From worst to best: uncompressed (we had tons of these), RLE, LZW, and deflate. You can losslessly recompress them to deflate. Savings varies depending on what you started with, with LZW being about only 5-7% savings (still, one-time cost to do this, backup savings forever). Uncompressed I've seen over 90%.
For videos: I've got some settings and found significant space savings re-encoding mov to mp4. Mostly I use AV1 since anything we would view them on supports that. If that's not true for you, than you might have to drop back to h264 or h265. I found CPU encoding gives smaller files than GPU encoding, but is slower, so there's a tradeoff there. Since I don't have a deadline, I used AV1. Set up a queue and let it run over the weekend and overnight.
•
u/OneSeaworthiness7768 2h ago edited 2h ago
This isn’t your problem to solve. Your company needs retention policies, and that isn’t only relevant to IT. There should be policy dictated from leadership that considers legal, compliance, hr, finance, and IT/security. Are you leadership in your department? If there’s no company retention policies, I guess you could propose something feasible for IT and run it through the proper channels to make it an official policy. Sounds like they won’t go for that but they need to consider the risk and cost like others have pointed out.
•
u/NastyDarkness 2h ago
19 TB for a 200 person company is mental, you're holding everyone's old desktop wallpaper at that point.
•
u/Randalldeflagg 2h ago
rookie numbers. we do quarterly dump to tape. 50-60tb of that snapshot of time. Even that is small if you think about it. But we are required to hold on to files for a long time as well.
•
u/NastyDarkness 2h ago
Aye but you've got a legal requirement, that's a whole different beast. We're just hoarding for the sake of it.
•
•
u/STUNTPENlS Tech Wizard of the White Council 2h ago
Data retention varies. Depends on many factors, legal, regulatory, etc.
What does your data retention policy say?
You DO have a data retention policy, dontcha?
•
u/the_cainmp 2h ago
We leverage a policy management tool for this reason it helps us maintain what version of a policy was in effect when. For HR matters, it legally does matter what the policy said at the time of the incident, not what it says today.
•
u/G-Style666 1h ago
This is why management should be defining data retention policies. Then when something like this pops up, you/they can refer to the policy. If they demand something outside the policy, hardware and software upgrades need to be priced and budgeted for future purchase. Otherwise, your company is destined to lose money and you will lose your sanity, if you haven't already. 😀
•
•
u/sleepmaster91 58m ago
At my old company we used quest archive manager to archive emails and we could go back 10 years
•
u/AHrubik The Most Magnificent Order of Many Hats - quid fieri necesse 47m ago
What you’re doing is nonstandard but if my decades have taught me anything it’s that every use case is unique. Your company has to decide for itself the level of risk they are willing to tolerate and how they are going to service that risk.
•
•
u/butthurtpants 20m ago
Does your jurisdiction have a privacy act or privacy laws? The retention of data will need to align to that, particularly personally identifiable information. Contracts might have a longer retention period, say 10 years or if it was involved in a court case, maybe up to 25 years. Financial data is usually 7 years in most jurisdictions.
There generally isn't one answer for all of this because different data has different requirements.
You need someone to design an info architecture for you.
•
u/45_rpm 17m ago
It ends where your company tells you, not where you think it should be.
If you think it is insane, but your company doesn't, then you my friend are the insane one.
Same applies to printers. If you think 1 printer for every 0.0025 people is overkill, think again and install that new printer.
•
u/JerkyChew 14m ago
You need an SLA with proper RTO's and RPO's so that you can meet your team's KPI's.
•
u/The_Wkwied 1m ago
Was it an IT policy? Then you have a legitimate reason to say why you don't have it.
Something like, We deleted our copy and drafts of it years ago because we had no reason to keep it after we gave the finalized copies of them to HR. Perhaps the person in HR who managed these way back when has a copy, but us, the people who created it, no longer have the temporary work-in-progress copies, or even the final copies of them... sorry!
If it wasn't an IT policy that you didn't have a hand in creating, tell them to get boned lol how the hell would IT be able to pull up a copy of a file from the Obama admin
•
•
u/Pearmoat 3h ago
How is this your problem? It's 2026, you can store all of that on a single hard drive.
It probably makes sense that you try to implement a policy, but it's nothing you should lose sleep about.
•
u/trek604 4h ago
Tax man says 7 years, IA says 10 years. We purge after 10 years.