How Amazon S3 Grew from Simple Storage into the Cloud Storage Standard

When something changes gradually, step by step, you usually don't notice the change, but after a couple of years, you can no longer imagine life without the results of those changes. The same happened with the internet.

The TV series The IT Crowd provides an excellent description of how the Internet worked during its early stages of development. One of the characters describes having to wait an entire night just to view a single photograph. Today that is hard to imagine. The internet is part of everyday life: it is where we work, shop, look things up and relax.

Changes like this are the work of countless scientists, engineers and technical specialists, and a significant share of that work lives in Amazon S3. Its 2006 interface is now implemented by storage providers around the world, including in the EU. Plenty has been written about S3, but for its 20th anniversary AWS engineers published many new details, and they are worth collecting in one place.

File Storage Before 2006

Before the emergence of Amazon S3, website files were typically stored on the disks of the same server running the website itself, and as long as there were few photos and archives, disk space was sufficient. When a disk filled up, a company would buy another server or separate network storage, usually buying enough capacity for a year or two ahead, because delivering and installing new hardware took weeks.

That headroom was only a rough estimate, so if a website suddenly became popular, free space would run out before new disks arrived, and administrators had to urgently delete old files or move part of the files to neighboring servers. The same administrators monitored disk health and backups; in small teams, this work usually fell to the developers. From 2006 on, files could instead be uploaded to Amazon over HTTP and paid for as you go, with no upfront purchase.

What Happened to S3 Over Twenty Years

S3 turned twenty on March 14, 2026, and for the anniversary AWS published a retrospective showing how modest the service was at launch. In 2006, the whole system fit into fifteen racks distributed across three data centers. Four hundred storage nodes provided a total capacity of approximately one petabyte, and the total throughput was 15 gigabits per second. The maximum size of a single object was limited to 5 GB, storing one GB cost 15 cents per month, and the system was designed for tens of billions of objects.

Today, S3 holds more than 500 trillion objects with a total volume of hundreds of exabytes, and the service handles more than 200 million requests per second across 123 Availability Zones in 39 regions. The maximum object size was increased to 50 TB in December 2025—a ten-thousandfold increase compared to 2006—and standard storage costs just over two cents per GB, about 85 percent cheaper than at the start of the project.

Key S3 metrics in 2006 and 2026 according to the AWS blog and the anniversary retrospective.

The boldest claim in the retrospective is that the code written for S3 in 2006 still works today without changes, even though several generations of disks and storage systems have been replaced and the request processing within the service has been completely rewritten. The claim largely holds: the closest compatibility came to breaking was in 2019.

At that time, AWS announced it would abandon one of the two methods of bucket addressing, and a year and a half later, it postponed the abandonment without announcing a new date. The S3 interface is implemented by many third-party storage providers today. Therefore, those who have never opened the AWS console also deal with the legacy of S3, as the protocol's limitations and features come with any compatible storage.

How S3 Started at Amazon

Jeff Barr holds the title of Chief Evangelist at AWS. A small clarification is needed here, as this position is not related to religious preaching but stems from the history of Macintosh marketing. In American tech companies, this is what they call a person who talks to developers about products. Barr has been doing this since 2003, and for twenty years, he has maintained the official AWS blog.

In 2019, Barr mentioned that Jeff Bezos's original assignment for S3 was very short: to create "malloc for the internet"—that is, an analog of the memory allocation function from the C language, but on a network scale. No other sources with this exact phrasing were found, and it is not repeated in AWS's anniversary materials.

We have to rely on the memory of one participant, albeit a very well-informed one. However, the comparison fits well with the first version of S3, since the malloc function allocates memory and returns a pointer to it without knowing what the program will put there, and S3 initially handled objects in almost the same way, building its operations around two operations: PUT for writing and GET for reading.

Development, according to Mai-Lan Tomsen Bukovec, who is now responsible for Data and Analytics at AWS and has been part of the S3 team since 2013, truly began in 2005. The service was released on March 14, 2006, as the first of the major AWS services. The EC2 compute service appeared in beta only in August of that same year, and Jeff Barr's launch post, as recalled in the anniversary retrospective, was without code examples or demonstrations.

The memories of the launch participants allow us to debunk the popular myth that Amazon simply started renting out servers that were idling between holiday sales. Werner Vogels, CTO of Amazon, wrote as early as 2011 that this story is a myth. According to him, AWS was considered a separate business from the start that could eventually grow to match the size of Amazon's retail business, and Amazon.com's spare capacity would have only lasted two months for the new business after launch.

How S3 Object Storage Differs from a File System

Almost everything that appeared in S3 later was built on top of the first two operations. An object in S3 is a sequence of bytes with metadata stored in a bucket and accessible via a key; the storage has no other structure. There are no directories in general-purpose buckets. The forward slash in the key logs/2026/11/05.gz remains a regular character, and the "folders" shown in the console are the result of a LIST request with a prefix and a delimiter.

Because of this model, an object in a general-purpose bucket cannot be partially modified like a regular file; any edit requires rewriting the entire object. You also cannot rename an object with a single request in a general-purpose bucket. First, the object is copied under a new key, and then the old object is deleted, so "moving a directory" with a million files actually means a million copy requests and a million delete requests. DELETE requests in S3 are free, but every copy is charged as a separate request. An exception is the S3 Express One Zone directory buckets, where, since June 2025, an object can be renamed with a single RenameObject request without moving data.

The "S" in the name stands for Simple, though the service only looks simple from the client's perspective. Andy Warfield, Vice President and Distinguished Engineer at S3, describes the internal structure of the service as hundreds of microservices that form four major parts: a fleet of ingress servers with a REST interface, a namespace service, a hard disk storage fleet, and a group of servers for background tasks like replication. Each major part has its own lead and teams, and interaction between teams is formalized at the API level, just like interaction between services.

S3 Evolution, 2006–2026

In its early years, AWS published object counters almost quarterly, and the growth is clearly visible in Jeff Barr's posts on the official blog. By the end of 2006, S3 held 2.9 billion objects; by the end of 2010, it had 262 billion with a peak load of over 200,000 requests per second. It took six years to reach the first trillion, hitting it in June 2012, and the second trillion was reached in less than a year, by April 2013. In 2022, there were more than 200 trillion objects, and the average load exceeded 100 million requests per second. Comparing the end of 2006 to the spring of 2026, the number of objects has grown approximately 170,000 times, and the peak load of 2010 versus the current 200 million requests per second differs by about a thousand times.

AWS News Blog (2006–2010, 2011, 2012, 2013, 2022, 2023), About Amazon and AWS Anniversary Retrospective.

Until July 2018, the documentation advised adding a random hash to the beginning of the key so that requests would be distributed across different index partitions. Along with increasing the request rate limits, AWS revoked this recommendation, though keys with a hash at the beginning are still found in older projects.

Where Strong Consistency Came From

Until December 2020, the service did not guarantee that reading immediately after a rewrite would return the new version of the object, or that a LIST request would show the newly written key. Werner Vogels, in a 2021 article, explains this behavior through the design of the metadata subsystem. A cache sits in front of it, designed to ensure requests are fulfilled even if one of its parts fails. Because of this, a write might occasionally pass through one part of the cache, while a read is served by another part that hasn't received the new version yet. Vogels calls this cache behavior the primary source of S3's eventual consistency.

Before strong consistency was available in S3, clients had to achieve it using their own means. For example, Netflix released s3mper, which used DynamoDB alongside S3 as a consistent store to catch rare instances of stale responses, and in the Hadoop ecosystem, S3Guard solved a similar problem. For data lakes, which were being built massively on top of S3 from the mid-2010s, this was a very practical issue, as a job that failed to see part of the newly written files would read incorrect results without any errors.

The simplest solution would have been to bypass the cache and go directly to the storage layer, and the team considered this as one of the first options, but rejected it because the service would run slower without the cache. The solution was built on replication logic that was already at the metadata storage level and allowed for establishing the order of operations for each object.

A new component called a "witness" was added to the replication logic. The witness receives a notification for every object change, and upon a read, the cache first asks the witness if the object has changed; if it has, it fetches the fresh data from the storage layer.

Mai-Lan Tomsen Bukovec calls the basis of the solution a "replicated log"—a chain of nodes through which every write passes in order and receives a sequence number. Based on the description, it sounds like the same replication logic, though Tomsen Bukovec does not state this explicitly. Building the log, according to her, took a long time, and she calls it the "secret sauce of S3."

Strong consistency required additional hardware, and according to Tomsen Bukovec, the team debated whether to pass these costs on to customers, but ultimately decided not to. In December 2020, strong consistency became available for all buckets immediately, without a separate storage class and without extra charge.

The implementation also had to be verified; as Vogels writes, the verification methods required more work than the code itself, because the team needed a system that was "provably correct, not just probably correct." Verification continues today, and the anniversary retrospective states that with every change to the index subsystem code, automated proofs confirm that consistency is not violated.

S3 does not provide transactions even today. If two clients simultaneously write an object with the same key, the one S3 considers last will remain; both clients will receive a successful response, and neither will know about the data loss. AWS added protection against this situation only in August 2024 in the form of conditional writes with the If-None-Match: * header. With this header, an attempt to create an object with an already occupied key receives a 412 Precondition Failed response. In November 2024, a write with theIf-Match header was also introduced. It only succeeds if the object has not changed since it was read. Bucket configurations themselves, according to the consistency model documentation, remain eventually consistent, so a recently deleted bucket might still appear in a bucket list for some time.

How S3 Is Built Internally

All 500 trillion objects are ultimately written to standard hard disks, and the S3 architecture is easiest to understand if you know how unevenly these disks have evolved.

According to Warfield's calculations, since the first IBM RAMAC storage drive in 1956, disk capacity has grown 7.2 million times, while head positioning time has decreased only 150 times, because mechanics accelerate much slower than recording density grows. A modern disk provides about 120 I/O operations per second during random access, and this number does not depend on capacity.

In 2023, when Warfield wrote his text, the largest disks held 26 TB, and manufacturers promised to reach 200 TB; at that time, with uniform load distribution, there would be approximately one operation per second for every 2 TB of data. There is plenty of disk space, but the number of requests per second per terabyte stored decreases with each generation of disks. This gap explains almost everything in S3's design.

Growth of hard disk capacity and head positioning speed from 1956 according to Andy Warfield

In AWS, the distribution of requests across disks is called "heat management," and disk heat is measured by the number of requests arriving at it simultaneously. If several disks become overloaded, requests to them queue up, latency is passed to the layers working on top of the disks, including metadata lookup and erasure coding, and soon the entire service responds more slowly. S3's sheer scale helps manage the heat.

An individual client loads the storage unevenly, as they may not access it for a long time and then send a burst of requests; however, when there are millions of such clients, the bursts of one coincide with the quiet periods of others, making the overall flow so smooth that no single client can significantly change it.

Erasure coding, which exists primarily for durability, also helps smooth the load. An object is split using Reed-Solomon code into k data parts and m parity parts, and any k parts out of k+m are sufficient for reading, so a request can be routed away from a busy disk. Different objects in the same bucket are intentionally spread across different sets of disks.

According to Warfield, with tens of thousands of clients, buckets are distributed across millions of disks, and a burst from thousands of parallel Lambda functions analyzing genomes can be spread across more than a million drives at once. AWS notes in the 2026 retrospective that as S3 grows, individual client loads have less and less impact on each other, which increases reliability for everyone.

In addition to disks, the index can become overloaded, handling more requests than the storage system itself, as HEAD and LIST requests do not reach the data at all. The most famous S3 outage was also related to the index. On February 28, 2017, in the US-EAST-1 region, an engineer made a mistake in a command parameter and shut down more servers than intended, requiring an index restart that took longer than AWS expected. Therefore, as stated in the incident analysis, the plan to split the index into smaller cells, which was intended to start later that year, was implemented immediately.

The course of the February 28, 2017 outage in the US-EAST-1 region according to the incident analysis published by AWS. Pacific Time.

Below the index and erasure coding is a layer responsible for data on a specific disk, and this layer was rewritten from scratch in Rust, with the new implementation named ShardStore. Alongside the production code, the team maintains a simplified executable model of the same layer, also written in Rust, which occupies about one percent of the actual implementation's volume. With every code change, property-based automated tests check that the implementation behaves the same as the model, and the methodology is described here. The anniversary retrospective states that over eight years, the data movement component was also rewritten in Rust, and work on other areas affecting request processing speed continues.

Data integrity is checked constantly; special auditor services scan every byte on all disks and trigger recovery as soon as signs of corruption are detected. AWS explains that "eleven nines" of data durability is the design goal, which dictates the level of data redundancy and the capacity buffer for recovery. Tomsen Bukovec adds that thanks to these auditors, the company can provide actual data durability metrics for a week, a month, or a year.

In addition to auditors, data durability is protected by organizational procedures, specifically a separate review that every change capable of affecting data durability must undergo. This review is based on an information security threat model, where the author of the change lists all possible failures and describes what protection is already in place for each case. Preference is given to coarse, broad constraints that cover entire classes of risks.

Old Buckets and Third-Party Requests

The most famous S3 leaks occurred through buckets accidentally opened to the entire internet, and for a long time, the service itself did not prevent critical security mistakes, such as accidentally making a bucket public. Buckets were always created as private, but it was too easy to open them with a single checkbox or an ACL list; therefore, in 2018, AWS added Public Access Block, in 2021, the option to disable ACLs appeared, and since April 2023, both of these parameters are enabled by default for every new bucket, regardless of how it is created. This change did not apply to buckets created earlier, and in older accounts, they still exist with their original settings.

In the spring of 2024, developer Maciej Pocwierz created an empty bucket in the eu-west-1 region for a test project and, two days later, found a bill for over $1,300 for a single day. Nearly 100 million PUT requests came from installations of a popular open-source tool whose default configuration used the same bucket name as his. The storage rejected these requests with an access error, but the bucket owner was still charged for the rejected requests at that time. There was no way to protect against such a collision, as bucket names are global across all AWS clients, and anyone could create a bucket with a name used by a third-party program.

After the publicity, AWS stopped charging owners for 403 responses to requests coming from outside their account or organization; this change was rolled out between May and August 2024. This relaxation does not apply to requests from within the owner's own organization, and the bucket owner still pays for access denials for such requests. Protecting against such name hijacking became possible only in March 2026, when S3 introduced account-level namespaces with names like prefix-accountID-region-an. An existing bucket cannot be renamed into a new namespace, so the bucket must be recreated and the data migrated.

Request limit errors and region-locking issues occur less frequently but often have more severe consequences. S3 supports at least 3,500 write and 5,500 read requests per second per prefix, but during a sudden load spike, it will return a 503 Slow Down error until it can split the overloaded index partition. Dependence on a single region is also dangerous because alerting tools may fail along with the application. During the 2017 outage, AWS itself could not update its service status page until 11:37 because the console for that page depended on S3.

How the S3 API Became the Common Language of Object Storage

S3 XML responses still contain the namespace http://s3.amazonaws.com/doc/2006-03-01/ with the date of the first API version, and the exact same string appears in the documentation of storage providers that have nothing to do with Amazon. AWS states that the S3 interface became an industry standard and that many manufacturers release compatible systems. These include open-source projects like Ceph and MinIO, as well as cloud platform storage services.

For developers, compatibility is beneficial because AWS CLI, familiar SDKs, and libraries that know how to write to S3 will work with another storage provider simply by changing the endpoint address. Along with the interface, compatible storage services inherit solutions from 2006, such as a flat key space without directories, the lack of renaming, and full object overwrites, even if the underlying storage is structured differently than S3.

The cost of this role for AWS is evident in how the company tried to abandon one of the two methods of bucket addressing. A bucket name can be specified either in the request path, as in https://s3.amazonaws.com/bucket/key, or in the hostname, as in https://bucket.s3.amazonaws.com/key.

In the spring of 2019, AWS announced that it would completely end support for the first method, path-style addressing, on September 30, 2020. Customer feedback forced the company to reconsider the plan, and a week later, Jeff Barr admitted that the announcement was "quite quiet, actually even too quiet." He then announced that path-style addressing would be maintained for all buckets created no later than September 30, 2020.

The main argument for the AWS plan was that hostname-based addressing limits the circle of customers affected by an outage—what the company calls the "blast radius." Buckets with dots in their names hindered the plan. Such names are valid and widely used for hosting websites, but they cause known TLS certificate issues when using hostname-based addressing. In an update on September 23, 2020, AWS announced it was postponing the end of path-style addressing to give customers time to transition, and it has not announced a new date since.

Self-hosted S3-compatible storage without a separate domain for each bucket typically relies on path-style addressing, and SDKs automatically switch to this method when the endpoint address does not contain a domain name. So AWS’s claim that “2006 code works without changes” holds partly because the company never fully completed the move away from path-style addressing.

Compatibility, however, is never complete. Each provider supports its own subset of the S3 API: some methods and headers may be missing or behave differently, and some features are available only on request through technical support. The list of supported methods should therefore always be reviewed before migrating.

Our storage also has incomplete compatibility. Object versioning is supported, but it cannot yet be enabled via an API request or the management console, so users must contact technical support. There is currently no server-side encryption in the storage, and a single storage account can have no more than one hundred buckets. The object address is constructed via a path.

In January 2025, AWS itself broke compatibility with third-party storage service when it changed the behavior of its client libraries. Official SDKs and the AWS CLI in versions released around January 15 began to default to calculating CRC32 or CRC64NVME checksums for every upload and verifying them during download.

For S3 itself, the change is useful, but many third-party storage service did not understand the new headers and began returning errors to standard PutObject and UploadPart requests. Akamai, for example, lists the affected versions in its documentation, starting from AWS CLI 2.23.0 and boto3 1.36.0, and Outscale notes that in AWS CLI versions 2.23.0 to 2.23.4, the aws s3 cp command ignored the setting intended to disable checksum calculation. In the JavaScript SDK repository, AWS warns that it may enable new features by default without waiting for support from other companies' compatible storage services.

What S3 Is Becoming

In the last three years, S3 has gained features that do not align well with the word "Simple" in its name. In November 2023, S3 Express One Zone was introduced—a separate storage class with a new type of buckets called directory buckets. Data in them resides in a single selected Availability Zone, latency is measured in single-digit milliseconds, and the bucket name follows a special format with a zone identifier and the suffix --x-s3.

For the sake of speed, some standard functions were removed from directory buckets; at launch, for example, there was neither versioning nor replication, so migrating a bucket with all its settings to Express One Zone without verification is risky. AWS later reduced the prices for Express One Zone, cutting some line items by 85 percent.

Long before S3 Tables appeared, other companies like Snowflake were performing analytics on top of cloud data. In its 2020 IPO prospectus, Snowflake states that it offers its platform only in public clouds like AWS, Azure, and GCP—which compete with it—and that a "significant portion" of its business operates in the AWS cloud. The company also admits that its costs and gross margins depend on the prices negotiated with cloud service providers.

In 2024, AWS addressed this itself when S3 Tables arrived—managed tables in the Apache Iceberg format. The service manages these tables itself, specifically compacting small files into large ones and deleting outdated snapshots. At the same time, the S3 Metadata feature was released in preview, allowing users to search for objects by metadata using a standard table query without scanning massive buckets with LIST requests. In July 2025, it expanded to existing objects. In December 2025, S3 Vectors became generally available.

S3 Vectors stores vector representations for semantic search and Retrieval-Augmented Generation (RAG), and this feature is what primarily links S3 to artificial intelligence. Vector representations of text or images require space, and keeping hundreds of millions of vectors in a specialized database is expensive, so AWS offers to store vectors in S3 alongside the source documents. One S3 Vectors index holds up to 2 billion vectors and responds to queries in less than 100 milliseconds.

According to the retrospective, between July and December 2025, users created more than 250,000 indices, uploaded 40 billion vectors, and performed about a billion queries. For comparison, a billion queries over five months averages about 75 queries per second—less than one-millionth of the 200 million queries per second handled by S3 itself, though this five-month period includes the preview phase.

AWS states that S3 should become a universal foundation for any data that is stored once and processed in place, without moving it between specialized systems. Whether this vision will be fully realized is hard to say, as specialized vector databases and tabular stores outside of S3 continue to evolve on their own paths.

Conclusion

Almost all the figures in this article are taken from AWS publications, so they should be viewed as statements from an interested party. The memories of participants are also not an absolutely reliable source. For example, Mai-Lan Tomsen Bukovec cites the initial limit of object size as 5 TB, even though it was 5 GB in 2006. The phrase "malloc for the internet" is only known through Jeff Barr's retelling. Limits, naming formats, and SDK behaviors change several times a year. Storage classes and their prices, Glacier archival storage, cross-region replication, and encryption were left outside the scope of this article.

The S3 interface became a standard due to the simplicity of its first two operations, pay-as-you-go pricing, and twenty years of backward compatibility, which made it easier for third-party developers to replicate someone else's API than to invent their own. Object storage is structured differently than a file system, and anyone who treats a bucket like a disk pays for it with extra requests, meaning more money and more time.