Is it possible to have a download link for the big train and test files? It would be better than downloading the files locally then uploading again to wherever I want to use them
Here is another option. In one of the other competitions, someone was kind enough to give a few tips on how to approach competitions with large data sets.
In this way, you download the large data sets once, and then essentially create a symbolic link from your online colab notebook to the large datasets offline.
I inspired from https://stackoverflow.com/questions/25010369/wget-curl-large-file-from-google-drive
wget and curl can offer you great ways to download files.I personally google colab and it helps a lot.
for this competition you can click the download button and start to download files in local than cancel the download.In the download tab you take the link than you use curl:
!curl -c ./cookie -s -L "link" > /dev/null
!curl -Lb ./cookie "link" -o filename
just change link with the link needed and filename with the name of the file and I think it will work even outside of colab. Hope this help others because it took me a while to find out a solution.
You can start downloading the file but then cancel the download. If you go to your download history you can see the URL there and use that.
Thanks!
Here is another option. In one of the other competitions, someone was kind enough to give a few tips on how to approach competitions with large data sets.
In this way, you download the large data sets once, and then essentially create a symbolic link from your online colab notebook to the large datasets offline.
https://zindi.africa/competitions/uber-movement-sanral-cape-town-challenge/discussions/702
I inspired from https://stackoverflow.com/questions/25010369/wget-curl-large-file-from-google-drive
wget and curl can offer you great ways to download files.I personally google colab and it helps a lot.
for this competition you can click the download button and start to download files in local than cancel the download.In the download tab you take the link than you use curl:
!curl -c ./cookie -s -L "link" > /dev/null
!curl -Lb ./cookie "link" -o filename
just change link with the link needed and filename with the name of the file and I think it will work even outside of colab. Hope this help others because it took me a while to find out a solution.