ArtLung [Joe Crawford]

Web developer & user interface engineer
Tinkering with the web since 1996

email: joe@artlung.com · twitter: @artlung · (photo from instagram)
San Diego, California, USA

Amazon S3 1,000,000,000,000

Some years ago I started using Amazon S3 as a kind of CDN for images here on ArtLung.com. It’s worked out well for me.

Amazon S3 holds all KINDS of content. And they’re filling up with stuff. No less a personage then Jeff Bezos posted to the Amazon Web Services blog saying: Amazon S3 – The First Trillion Objects.

I’m pleased with the way S3 has allowed me to bump the hosting of larger objects to a low-cost, configurable platform, and look forward to more AWS usage in the future.

ten comments so far...

I have over 1 GB (maybe 2GB?) of material (probably 900MB from this and a few other sites, plus some other family video and other multimedia) and pay about 2 bucks a month. For the first 2 years the monthly fee was always under a dollar. I point at the images on my own site, but do Apache redirects to the S3 images. It’s had the best benefit for Leah’s site because she has some super-large images (~3-6MB) available to people and I didn’t want to disable that.

Yes. When I think about offloading my own files from Flickr to my own site, I dread the prospect of replicating all the different sizes. There’s nothing built in to allow for that. It’s pure storage.

Now, theoretically, you could do an EC2 instance which can interact with your S3 buckets and do that kind of thing, but that’s an order of magnitude more work.

Unless you find someone has created an AMI which mimics those services.

Right now I am paying for both Flickr and SmugMug, so this might save me some money. Do you use an FTP client to manage images or a dedicated application? Thanks.

A combination of tools. Under Windows and Mac I’ve used a Firefox extension: S3Fox, on Windows I’ve used Cloudberry Explorer for Amazon S3, and on the Mac I use Cyberduck. There are also libraries for very nearly any language and framework you can think of at aws.amazon.com/code. The good thing is it’s really cheap to fool around with and experiment. The way I organize things is I started pointing things at files.artlung.com by redirect, and then I do a further step and redirect to S3. The thinking was if I stopped using S3, I could regress to still hosting at files.artlung.com. I have not had to do that, though when I reorganize things I will turn off some sites using those urls using .htaccess Apache directives. There are also a number of WordPress plugins designed to offload file hosting to S3, but I’ve not gone that route because I did not get a sense they had been fully battle tested. I’d be more likely to trust those now, as they’re all in the 4 year old range at this point and have more track record.

Excellent stuff. One more question. Flickr and SmugMug have code built in to generate image files of different sizes. I’m guessing that an S3 customer would need to do themselves prior to uploading the file. Is that correct? Thanks.

I just went down the rabbit hole investigating this. Seems a lot of people favor building the various sizes and not resizing on the fly.

thanks!

MAS directed me to this post, and I have a quick question. I’m working on a new site for my new company, and we’re going to eventually host a lot of images from events. The thing is, we need to be able to organize them into galleries, etc. I have used SmugMug for years, but can’t find any decent way to incorporate those galleries into a WordPress site.

However, I did find a pretty slick plugin called NextGEN Gallery that allows one to make galleries inside of WP, manage them, place them in post or pages, etc. But of course it wants to store the images on the site itself.

Now, my current hosting service allows for unlimited storage and bandwidth, and so far has been excellent as far as speed, etc. All things being equal, is there a benefit to storing those files in the cloud? I was willing to do that to get the management benefits I get from SmugMug, but as just pure file storage, it seems like I should just store them at my site.

I’m not familiar enough with SmugMug to WordPress integration to know if there’s a “better way” to do that cleanly, so won’t comment there.

I can’t speak to NextGEN Gallery or how it works.

If you don’t have programming resources in-house, I would look at off-the-shelf tools like W3 Total Cache, which as I understand it takes some of the pain about hosting resources in S3 – mirroring your content, and then handling pointing at those cloud files in a smart way.

My experience is on the DIY side of things — sorry I can’t be more helpful about your specific needs. I’d say generically that the cloud is great to the extent you have bandwidth you need to move elsewhere.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.