Pass result of Julia `download` to memory instead of file?

Question

With Julia 1.6's download function, the typical behavior is to output to a file. How can I save the result directly to something in memory?

E.g. I'd like something like:

result = download(url)
contains(result,"hello")

Answer 1

As suggested by the help text for download, use the Downloads library; download can take an IOBuffer. Example:

result = String(take!(Downloads.download(url,IOBuffer())))

Answer 2

Julia uses the curl library, or something similar, for the download function, and that library writes to a file by default, or to stdout, not to a C or Julia string. Consider that many downloads may be large, perhaps larger than system RAM, to see why.

You could easily extend Julia to download, create a string for the download, and remove the temp file:

import Base.download

function Base.download(url::AbstractString, String)
     tmpfile = download(url)
     str = read(tmpfile, String)
     rm(tmpfile)
     return str
 end

Watch out for big files though :)

Answer 3

You can also use UrlDownload.jl library. It uses HTTP.jl instead of curl, so it always keep the result in memory, and you can process it on the fly.

julia> using UrlDownload

julia> url = "https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/ext.csv"
"https://raw.githubusercontent.com/Arkoniak/UrlDownload.jl/master/data/ext.csv"

julia> urldownload(url, parser = x -> String(x))
"x,y\n1,2\n3,4\n"