'Using curl in a bash script and getting curl: (3) Illegal characters found in URL
So I have a very simple bash script that is curl'ing to an auth server for a header. The header url is written to a var and then used in the next curl call. When using the var set in the first curl call I am getting "curl: (3) Illegal characters found in URL". I am able to echo the var and all looks good, I am even able to reset the var (in my example below) and it works.
The Bash script
URL=$(curl -i -X GET -H "X-Auth-User: MyUserna,e" -H "X-Auth-Key: MyAPIKey" "https://urlToAuthServer.tld/auth/v1.0/" | grep "X-Storage-Url:" | awk '{print $2}')
curl -X GET -H "X-Auth-Token: MyAuthTok" "${URL}/folder/myfile.txt" -o ./myfile.txt
When running the above example I get:
curl: (3) Illegal characters found in URL
The URL var looks like this (no illegal chars)
https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534
When I do this in terminal it works:
$ URL=$(curl -i -X GET -H "X-Auth-User: MyUserna,e" -H "X-Auth-Key: MyAPIKey" "https://urlToAuthServer.tld/auth/v1.0/" | grep "X-Storage-Url:" | awk '{print $2}')
$ echo $URL
https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534
Now I copy and paste the string and reasign it to URL like so (again all in terminal):
>$ URL="https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534"
>$ curl -X GET -H "X-Auth-Token: MyAuthTok" "${URL}/folder/myfile.txt" -o ./myfile.txt
It works.
So why am I getting the "curl: (3) Illegal characters found in URL" error in the first example?
Update
I ran this:
printf %s "$URL" | xxd
Here is the output (addressed changed up you get the idea)
0000000: 6874 7470 733a 2f2f 6461 6c30 352e 6f62 https://server.ob
0000010: 6a65 6374 7374 6f72 6167 652e 736f 6674 jectstorage.lite
0000020: 6c61 7965 722e 6e65 742f 7631 2f41 5554 sabers.com/v1/AUT
0000030: 485f 6665 3235 3339 3434 2d38 6433 322d H_aE2563981-7d32-
0000040: 3432 3138 2d61 6566 632d 6665 6638 3465 4201-bdoi-fef94a
0000050: 6166 3331 6232 0d ag11c8.
Solution 1:[1]
The $URL contains a \r
(CR) at the end (0d
). Remove it with
URL=${URL%$'\r'}
before using it with curl
.
Solution 2:[2]
I had the same problem, but it was caused by me having created the script through Windows, which meant that the end of line had an unnecessary \r.
A simple dos to unix conversion fixed it.
dos2unix <scriptname>
Solution 3:[3]
stripping away trailing \r
tr -d '\r' < test.sh > testWithoutR.sh
Solution 4:[4]
It could be an enter at the end of the file. Try to remove it or delete it, note that the enter character is not something view-able....
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | choroba |
Solution 2 | ianiver9 |
Solution 3 | |
Solution 4 | jeff porter |