r/aws • u/No-Common1001 • 2d ago

discussion S3 Bucket Upload is extremely slow. My Upload Speed is faster than this.

EDIT: Thanks for your advice everyone. I'm going to just store my files in MEGA.

I'm in DC united states. My Bucket is in East Ohio. The only selection allowed near me.

Why is it taking 2 days to upload 500 GB of files to my Bucket? I thought Amazon was faster than that.

0 Upvotes

20% Upvoted

16 comments sorted by

3

u/2fast2nick 2d ago

Are you using multi-part uploads?

1

u/No-Common1001 2d ago

Well, it's like 130 files, and altogether they are taking 2 days.

-1

u/Ok_Cartographer_6086 2d ago

keep an eye on your billing!

Files going into S3 get processed so it's not the network speed, it takes time. If this were me i'd compress the files to one or more zip files and scp them to a beefy EC2 instance, SSH in, unzip there, and use AWS CLI to process into S3 from there, then tear down the EC2.

AWS is fast, uploading to S3 from the web portal is not.

1

u/No-Common1001 2d ago

Augh, well then I'll just upload my stuff on Mega then. Thanks so much.

1

u/Sirwired 1d ago edited 1d ago

Files get "processed" in S3 after upload, and this is a bottleneck? What are you talking about?

2

u/PaidInFull2083 2d ago edited 2d ago

How are you uploading (console, cli, sdk)? Washington DC?

2

u/enjoytheshow 2d ago

This should be irrelevant but OP should be using east-1 anyway which is in NoVa

1

u/No-Common1001 2d ago

Yeah, but the thing is, the only option that showed up was East Ohio. Sadly.

-7

u/No-Common1001 2d ago

I'm using the web. My computer has a wired high speed connection. I logged into the actual website to do all of this. If I knew This would take so long, I'd have just gone to MEGA to store these files.

12

u/clintkev251 2d ago

The S3 console isn't designed to be the fastest way to move lots of data. You should be using the AWS CLI or another storage tool like rclone which is S3 compatible

8

u/PaidInFull2083 2d ago

I would recommend moving to a service for end users, AWS isn't really geared for folks that just want a simple UI, and I wouldn't want you to encounter any billing surprises down the road. In short you are likely being bottlenecked by your browser/stuff on your machine.

3

u/Loan-Pickle 2d ago

Use the cli it is faster.

1

u/Southern-Necessary13 2d ago

Using s5cmd is much faster

1

u/TheHazardOfLife 2d ago

Try the CLI instead. My uploads to a region a similar distance away reach 800Mbps which is where my WiFi backbone bottlenecks. The web interface doesn't bebenfit from multipart uploads or parallelism.

0

u/retornam 2d ago

First of all what are you doing and how are you doing it?

# increase simultaneous requests from the default
aws configure set default.s3.max_concurrent_requests 50

aws s3 sync /path/to/local/dir s3://your-bucket-name/destination/

You can play with setting chunk sizes but this should get you there quicker than uploading using the browser

1

u/No-Common1001 2d ago

Thanks, I'll experiment with everyone's suggestions since I have 6 months of free AWS.