Skip to content

Add create to Storage that takes an InputStream #172

Closed
Closed
Add create to Storage that takes an InputStream#172
Assignees
aozarov
Labels
api: storageIssues related to the Cloud Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Description

@mziccard
mziccard
opened on Sep 22, 2015

Storage class now has a method to create an object:

create(BlobInfo blobInfo, final byte[] content, BlobTargetOption... options)

I know that we already provide a BlobWriteChannel but it might be useful to also have:

create(BlobInfo blobInfo, InputStream content, BlobTargetOption... options)

for uploading small files without reading their bytes. Thoughts?

Activity

  1. mziccard
    added
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    api: storageIssues related to the Cloud Storage API.
    on Sep 22, 2015

  2. aozarov commented on Sep 22, 2015

    @aozarov
    aozarov
    on Sep 22, 2015
    Contributor

    I don't like having various flavors of the same method/functionality, especially when the difference is so subtle because at some point it makes the API less readable (what about adding also ByteBuffer option,....). This method should be used when content can live in memory. I picked bytes[] because I thought it is a good common denominator but could be convinced to pick something else.

    It is fairly simple to convert a InputStream to byte[] (e.g. by using Guava).

  3. mziccard commented on Sep 22, 2015

    @mziccard
    mziccard
    on Sep 22, 2015
    ContributorAuthor

    I see your point but let me play the devil's advocate here.
    It could argued that having both methods makes the library easier to use, as for instance, a user with a small FileInputStream is not required to read the stream to a byte array (even if it is farly simple) but can upload that with a single library call.

  4. aozarov commented on Sep 23, 2015

    @aozarov
    aozarov
    on Sep 23, 2015
    Contributor

    I assume that your suggestion includes also using a different RPC (apiary with an inputstream for media uploader - see here) rather than us reading the content into a byte array and using the same rpc?

    It would make more sense in that case even though I think generally we should recommend using resumable uploads for big files (via BlobWriteChannel) and non-resumables for small ones. Can you check if current implementation of the MediaUploader will really chunk when using direct/non-resumable upload rather than itself reading all in memory before sending the request?

  5. mziccard commented on Sep 23, 2015

    @mziccard
    mziccard
    on Sep 23, 2015
    ContributorAuthor

    Yes I was planning on adding a different RPC which makes use of InputStreamContent (see here) rather than ByteArrayContent.

    From the docs of MediaHttpUploader I read:

    For resumable uploads, when the media content length is known, if the provided InputStream has InputStream.markSupported() as false then it is wrapped in an BufferedInputStream to support the InputStream.mark(int) and InputStream.reset() methods required for handling server errors. If the media content length is unknown then each chunk is stored temporarily in memory. This is required to determine when the last chunk is reached.

    Which makes me think that is always tries to use resumable uploads (even for ByteArrayContent).

  6. aozarov commented on Sep 23, 2015

    @aozarov
    aozarov
    on Sep 23, 2015
    Contributor

    Yes, unless we pass true to setDirectUploadEnabled. see here. So my understanding (but I didn't dig deep) is that using this apiary client resumable downloads can work in chunks (for all input types) but for non-resumable (which is what I think we want here) one need to specify direct-upload and I was not sure if that will not load the complete content to memory before sending it (conceptually it does not have to). Can you verify these assumptions?

  7. mziccard commented on Sep 23, 2015

    @mziccard
    mziccard
    on Sep 23, 2015
    ContributorAuthor

    When directUploadEnabled is set to true, whether data are read in chunks or not depends on the type of XXXContent. In the case of all subclasses of AbstractInputStreamContent data reads happen in chunks of 2048 bytes.

  8. aozarov commented on Sep 23, 2015

    @aozarov
    aozarov
    on Sep 23, 2015
    Contributor

    You mean this AbstractInputStreamContent? As this
    one seems to write all in one write... (or do they override something?)

  9. mziccard commented on Sep 23, 2015

    @mziccard
    mziccard
    on Sep 23, 2015
    ContributorAuthor

    I might be wrong but doesn't AbstractInputStreamContent use IOUtils.copy which then calls BufferStreams.copy? That should work in chunks (4096B actually).

  10. aozarov commented on Sep 23, 2015

    @aozarov
    aozarov
    on Sep 23, 2015
    Contributor

    Ok, if you are sure that the destination (out) is streamed when directUploadEnable is true (which would make sense) rather than buffered and sent as one blob then I think your proposal of adding this method (and an extra RPC) make sense. Maybe we can use one RPC with InputStream then....

  11. aozarov commented on Sep 30, 2015

    @aozarov
    aozarov
    on Sep 30, 2015
    Contributor

    Implemented by #186

  12. aozarov
    closed this as completedon Sep 30, 2015
  13. JustinBeckwith
    assigned
    aozarov
    on Feb 1, 2021
  14. github-actions
    added a commit that references this issue on Jun 21, 2022

    chore: Integrate new gapic-generator-java and rules_gapic (#172)

    2d9925d

  15. 4 remaining items

  16. github-actions
    added a commit that references this issue on Jul 18, 2022

    build(deps): update dependency org.apache.maven.plugins:maven-project…

    2e0d82a
  17. github-actions
    added 7 commits that reference this issue on Oct 4, 2022

    deps: update dependency com.google.cloud:google-cloud-shared-dependen…

    e561188

    deps: update dependency com.google.cloud:google-cloud-shared-dependen…

    e3dd2dd

    chore(main): release 1.2.6 (#173)

    03cc152

    chore(main): release 1.1.7 (#187)

    f4d027a

    chore(main): release 1.3.0 (#172)

    f6cf47b

    chore(main): release 0.6.2 (#173)

    09e986f

    chore(deps): update dependency com.google.cloud:google-cloud-vmmigrat…

    58732f9
  18. github-actions
    added a commit that references this issue on Nov 8, 2022

    chore(java): add a note in README for migrated split repos (googleapi…

    e5eb1a4
  19. meltsufin
    added a commit that references this issue on Dec 22, 2025

    chore(deps): update dependency com.google.cloud:libraries-bom to v5.7…

    a6052b3
  20. chingor13
    added a commit that references this issue on Jan 22, 2026

    chore(deps): update dependency com.google.cloud:google-cloud-datastor…

    e155100
  21. suztomo
    added a commit that references this issue on Mar 11, 2026

    chore(deps): update dependency com.google.cloud:libraries-bom to v5.7…

    106321b
  22. suztomo
    added a commit that references this issue on Mar 23, 2026

    tests: run integration tests against emulator (googleapis#172)

    597d95d
  23. suztomo
    added a commit that references this issue on Mar 30, 2026

    chore(deps): update dependency com.google.cloud:libraries-bom to v8.1…

    c43811b
  24. lqiu96
    added a commit that references this issue on Apr 1, 2026

    chore(deps): update dependency com.google.cloud:libraries-bom to v8.1…

    aeb254a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
    Sponsor
    SponsoredKunjungi sekarang
    Promo