AWS Simplifies Data Management with S3 Annotations and Materialized Views
Amazon Web Services (AWS) has introduced annotations, a new metadata capability that lets users attach business context directly to objects in Amazon Simple Storage Service (S3). This feature aims to simplify data management and reduce complexity by providing a single source of truth for metadata. With S3 annotations, users can add up to 1,000 named annotations per object, each up to 1 MB in size, in flexible formats like JSON, XML, or YAML.
The annotations can be modified or deleted as needed, and they share the same durability and consistency properties as the object. Retrieval performance is similar to S3 Standard storage with no additional charges for storage retrieval. Users manage annotations through a set of new S3 APIs.
AWS has also introduced materialized views in AWS Glue using Spark, which store precomputed query results. This feature enables fast and efficient querying of large datasets by reducing query time by up to 93% and data scanned by 99%. Materialized views can be incrementally refreshed as underlying data changes.