Repository navigation
Add create to Storage that takes an InputStream #172
Description
Activity
I don't like having various flavors of the same method/functionality, especially when the difference is so subtle because at some point it makes the API less readable (what about adding also
ByteBufferoption,....). This method should be used when content can live in memory. I pickedbytes[]because I thought it is a good common denominator but could be convinced to pick something else.It is fairly simple to convert a
InputStreamtobyte[](e.g. by using Guava).I see your point but let me play the devil's advocate here.
It could argued that having both methods makes the library easier to use, as for instance, a user with a smallFileInputStreamis not required to read the stream to a byte array (even if it is farly simple) but can upload that with a single library call.I assume that your suggestion includes also using a different RPC (apiary with an inputstream for media uploader - see here) rather than us reading the content into a byte array and using the same rpc?
It would make more sense in that case even though I think generally we should recommend using resumable uploads for big files (via BlobWriteChannel) and non-resumables for small ones. Can you check if current implementation of the MediaUploader will really chunk when using direct/non-resumable upload rather than itself reading all in memory before sending the request?
Yes I was planning on adding a different RPC which makes use of
InputStreamContent(see here) rather thanByteArrayContent.From the docs of
MediaHttpUploaderI read:For resumable uploads, when the media content length is known, if the provided InputStream has InputStream.markSupported() as false then it is wrapped in an BufferedInputStream to support the InputStream.mark(int) and InputStream.reset() methods required for handling server errors. If the media content length is unknown then each chunk is stored temporarily in memory. This is required to determine when the last chunk is reached.
Which makes me think that is always tries to use resumable uploads (even for
ByteArrayContent).Yes, unless we pass
truetosetDirectUploadEnabled. see here. So my understanding (but I didn't dig deep) is that using this apiary client resumable downloads can work in chunks (for all input types) but for non-resumable (which is what I think we want here) one need to specify direct-upload and I was not sure if that will not load the complete content to memory before sending it (conceptually it does not have to). Can you verify these assumptions?When
directUploadEnabledis set to true, whether data are read in chunks or not depends on the type ofXXXContent. In the case of all subclasses ofAbstractInputStreamContentdata reads happen in chunks of 2048 bytes.You mean this
AbstractInputStreamContent? As this
one seems to write all in one write... (or do they override something?)I might be wrong but doesn't
AbstractInputStreamContentuseIOUtils.copywhich then callsBufferStreams.copy? That should work in chunks (4096B actually).Ok, if you are sure that the destination (
out) is streamed whendirectUploadEnableis true (which would make sense) rather than buffered and sent as one blob then I think your proposal of adding this method (and an extra RPC) make sense. Maybe we can use one RPC with InputStream then....- added a commit that references this issue
on Jun 21, 2022 4 remaining items
- added a commit that references this issue
on Jul 18, 2022 - added 7 commits that reference this issue
on Oct 4, 2022 - added a commit that references this issue
on Nov 8, 2022 - added a commit that references this issue
on Dec 22, 2025 - added a commit that references this issue
on Jan 22, 2026 - added a commit that references this issue
on Mar 11, 2026 - added a commit that references this issue
on Mar 23, 2026 - added a commit that references this issue
on Mar 30, 2026 - added a commit that references this issue
on Apr 1, 2026
Storageclass now has a method to create an object:I know that we already provide a
BlobWriteChannelbut it might be useful to also have:for uploading small files without reading their bytes. Thoughts?