shim-downloder: retry logic

[vdice]

We have seen cases where the first attempt to download a shim binary by the downloader component fails (eg as helpfully pointed out in #464 (comment)). This will cause the install job pod to Init:Error and try again, eventually succeeding assuming the underlying issue is temporary.

It may be better to add retry logic to the downloader component such that it retries the download in a TBD manner (eg N number of retries, perhaps with a global timeout, etc)

Open questions:

1. What retry defaults should be used?
2. Should any/all of the retry configuration be exposed to the human operators (eg via chart values or potentially the Shim CRD)?

[Shunpoco]

@vdice
Could you assign me?

My two cents for your questions:

1. I think we can start from some random number (e.g., 3 retries, and 2 seconds sleep for each?)
2. I don't think we should expose each configuration variables in CRD level as they are too much detailed, but how about allow users to specify ConfigMap for the configuration in Chart/CRD level?

[vdice]

@Shunpoco thanks! Sounds good on both points. Also glad to implement 1 (internal service concern) if we'd like to defer 2 until later. But for 2, the configmap seems like a reasonable way to go.