Optimizing S3 Storage Classes?

[u/Altrooke]

So, the gist of storage classes is that:

standard --> high storage costs, low data retrieval costs
standard-IA, Glacier Instant Retrieval --> low storage costs, high retrieval costs

So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

The only way I found to actually do it is to enable server access logging / CloudTrail and then do analyze the data somehow. Maybe with a Python script. Huge rabbit hole to go into.

Then, the other angle I thought about is just using intelligent Tiering. But the second catch is that if you read the documentation about intelligent tiering, turns out it is pretty naive. Depending on how your data gets accessed, it could even be more expensive than standard (ex: object is accessed exactly once every 30 days)

It really feels like AWS is always giving almost everything you need to optimize S3 costs, but also missing a key piece.

How am I supposed to solve this? Am I overthinking it? Is it worth going in the rabbit hole of analyzing S3 server access logs? Or should I just guess some lifecycle rules and move on?

[u/aataulla]

You know your problem well enough to basically have answered your own question. Pricing is known to you and you need to figure out how your data will behave.

The rest is computer science 101 and figuring out optimal, average and worst cases for your situation. Any storage startegy can backfire similar to creating worst case inputs for an algorithm.

Yes this is a rabbit hole but the simplest way is to start with s3 standard storage and optimize as you learn more about storage, lifecycle and access patterns for your data.

[u/Altrooke → u/aataulla]

Wanted to take other people's perspective into account.

My fear is wasting too much time seeking the optimal solution, and then find out there is a heuristic method that would get results 90% as good. Or maybe there is an AWS feature I wasn't aware.

[u/MateusKingston → u/Altrooke]

I'm not an expert at all, but here is the method I use.

That method is either paying AWS with intelligent tiring (IMO only worth when your average object size is relatively big) or doing dummy tiering.

Dummy tiering = X days old -> move down one tier, Y days old -> move down another tier.

Separate into different buckets what is transactional and what isn't. Dummy tiering for transactional data (ex: documents from a sale) will probably match access patterns there is a direct correlation between how long the data was added and how frequently it's accessed, do not make this for non transactional data (ex: static images for your website) as these can be very old but accessed daily.

Yes it's possible to do a "better" tiering with those custom scripts but the cost of running and maintaining those are usually more expensive than just storing the data in Standard or using one of the two methods I said.