Refactor Terratests

[nicogelders]

Right now the Terratests are a bit of a mess so I would like to refactor them by creating some basic methods that you can easily chain together to test the full setup.

The methods I suggest right now are:

• TerraformApplyAndDestroy : input testing.t and terraform.options; calls tf init, tf plan, tf, apply and defers tf destroy
• AssertContainersAreRunning: asserts if all containers in an ecs cluster are in the running status
• AssertAirflowWebserverIsReachable: Checks if you can access the airflow ui
• AirflowLogin: logs in to the airflow ui
• AssertAirflowLoggedIn: check if you are logged in to airflow
• AirflowAddDag: uploads a new dag to s3
• AssertAirflowDagExists: checks if a dag is visible in the UI
• AirflowRunDag: runs a dag
• AssertAirflowDagHasSucceeded: checks if a certain dag has successfully run via the UI

All these methods (except for TerraformApplyAndDestroy) make sure that airflow is deployed in a usable state and that you for example didn't mess up something in the configuration of airflow.

An example test would look like this (pseudo code):

func TestAirflowWithRBAC(){
   options = {
        "airflow_authentication": "rbac"
   }

    TerraformApplyAndDestroy(t, options)
    AssertContainersAreRunning("my-test-cluster", ["airflow-webserver", "airflow-scheduler"])
    AssertAirflowWebserverIsReachable("http://airflow.test")
    AirflowLogin("admin", "admin", "http://airflow.test") // only needed when rbac is enabled
    AssertAirflowLoggedIn("http://airflow.test")  // only needed when rbac is enabled
    AirflowAddDag("path/to/dag.py", "s3-bucket-name", "s3/path/key/dag.py", "http://airflow.test")
    AssertAirflowDagExists("name_of_dag", "http://airflow.test")
    AirflowRunDag("name_of_dag", "http://airflow.test")
    AssertAirflowDagHasSucceeded("name_of_dag", "http://airflow.test")
}