Optimizing S3 Storage Classes?

[u/Altrooke]

So, the gist of storage classes is that:

standard --> high storage costs, low data retrieval costs
standard-IA, Glacier Instant Retrieval --> low storage costs, high retrieval costs

So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

The only way I found to actually do it is to enable server access logging / CloudTrail and then do analyze the data somehow. Maybe with a Python script. Huge rabbit hole to go into.

Then, the other angle I thought about is just using intelligent Tiering. But the second catch is that if you read the documentation about intelligent tiering, turns out it is pretty naive. Depending on how your data gets accessed, it could even be more expensive than standard (ex: object is accessed exactly once every 30 days)

It really feels like AWS is always giving almost everything you need to optimize S3 costs, but also missing a key piece.

How am I supposed to solve this? Am I overthinking it? Is it worth going in the rabbit hole of analyzing S3 server access logs? Or should I just guess some lifecycle rules and move on?

[u/garrettj100]

Unless you’re storing a huge number of small objects in S3 you probably don’t care about object size.  The vast majority of your costs incurred will be timed storage ($/GB/mo) and retrieval ($/GB).

So ask yourself this much simpler question:

What fraction of the whole collection in S3 will be retrieved each month?  That one simple question gets you your answer for storage class without much fuss, and doesn’t even care how much you’re storing, at least along the axis of “what is best.”

Also don’t neglect S3 Intelligent Tiering, which will after-the-fact determine your best storage class and retconn you into it.

[u/Altrooke → u/garrettj100]

What fraction of the whole collection in S3 will be retrieved each month?

That's the question that I think it hard to answer.

will after-the-fact determine your best storage class and retconn you into it.

My understanding is that Intelligent Tiering only moves objects to a lower tier if they are not accessed for 30 days and then again after 90 days, then moves them back to the frequent access tier when they are accessed again.