Support S3 unsigned body #3323
Support S3 unsigned body #3323
Conversation
|
Detected 1 possible performance regressions:
|
gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Outdated
Show resolved
Hide resolved
richardwang1124
left a comment
richardwang1124
left a comment
There was a problem hiding this comment.
Nice! I think it looks good overall to me. Added a few comments.
gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Outdated
Show resolved
Hide resolved
gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Outdated
Show resolved
Hide resolved
gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Outdated
Show resolved
Hide resolved
gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Outdated
Show resolved
Hide resolved
gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Outdated
Show resolved
Hide resolved
This reverts commit 1692239.
| else | ||
| @encoded_buffer << "0\r\n#{trailer_string}\r\n\r\n" | ||
| @eof = true | ||
| break |
There was a problem hiding this comment.
nit: since the while loop has a condition on @eof, you probably don't need the break here
| CHUNK_SIZE = 1 * 1024 * 1024 # one MB | ||
| MIN_CHUNK_SIZE = 16_384 # 16 KB |
There was a problem hiding this comment.
I find these constant names a bit confusing - I think we should be more specific about what they are for?
If I'm understanding/remembering correctly, CHUNK_SIZE is the size for checksum/digest computation - so maybe CHECKSUM_CHUNK_SIZE and MIN_CHUNK_SIZE is being used as the default trailer chunk size? so maybe DEFAULT_TRAILER_CHUNK_SIZE
|
|
||
| full_chunk_overhead = @base_chunk_size.to_s(HEX_BASE).size + CHUNK_OVERHEAD | ||
| chunked_body_size = n_full_chunks * (@base_chunk_size + full_chunk_overhead) | ||
| chunked_body_size += partial_bytes.to_s(HEX_BASE).size + partial_bytes + 4 unless partial_bytes.zero? |
There was a problem hiding this comment.
nit, why hard code the 4 here? should it be CHUNK_OVERHEAD?
|
|
||
| # Wrapper for request body that implements application-layer | ||
| # chunking with Digest computed on chunks + added as a trailer | ||
| class AwsChunkedTrailerDigestIO |
There was a problem hiding this comment.
Wasn't really sure where to leave this comment - but this class is super complex. I know its indirectly tested through other things, but I think given the complexity we may want to add specs for it by itself to test different cases of length/chunk size
3 participants
This PR intends to improve performance for S3's
PutObjectandUploadPart.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
To make sure we include your contribution in the release notes, please make sure to add description entry for your changes in the "unreleased changes" section of the
CHANGELOG.mdfile (at corresponding gem). For the description entry, please make sure it lives in one line and starts withFeatureorIssuein the correct format.For generated code changes, please checkout below instructions first:
https://github.com/aws/aws-sdk-ruby/blob/version-3/CONTRIBUTING.md
Thank you for your contribution!