Create cluster success in UI but got ResourceErrorCode

[carolynhu]

Hi there, I encountered similar issue as this: #148

I'm using https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/master/examples/v2/gke/python/cluster.py and based on my recent changes #536 to create a GKE cluster.

My cluster is created successfully as least on the UI part.

But in the terminal, I got the following errors:

 Waiting for create [operation-1581380209089-59e41c59e1bfb-c62501e0-2a81feab]...
failed.
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1581380209089-59e41c59e1bfb-c62501e0-2a81feab]: errors:
- code: RESOURCE_ERROR
  location: /deployments/carolyn-manager/resources/carolyn-manager-3-cluster-py-type-apps
  message: "{\"ResourceType\":\"deploymentmanager.v2beta.typeProvider\",\"ResourceErrorCode\"\
    :\"DESCRIPTOR_URL_FETCH_ERROR\",\"ResourceErrorMessage\":\"location: /typeProviders/carolyn-manager-cluster-py-type-apps->$.descriptorUrl\\\
    n\\\"{\\\\\\\"originalResponse\\\\\\\":\\\\\\\"{\\\\\\\\\\\\\\\"paths\\\\\\\\\\\
    \\\\\":[\\\\\\\\\\\\\\\"/apis\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/apis/\\\\\\\\\\\
    \\\\\",\\\\\\\\\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\\\\\\\\\",\\\\\\\\\\\\\
    \\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/healthz\\\
    \\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/healthz/etcd\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"\
    /healthz/log\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"/healthz/ping\\\\\\\\\\\\\\\",\-bash: Waiting: command not found

Could someone explain why if there is an error my cluster can still be setup? Or is there any special requirements that I need to set to avoid such error?

[mmosca]

I am seeing the same error as well.

{"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/mycluster-cluster-type->$.descriptorUrl\n"{\"originalResponse\":\"{\\\"paths\\\":[\\\"/apis\\\",\\\"/apis/\\\",\\\"/apis/apiextensions.k8s.io\\\",\\\"/apis/apiextensions.k8s.io/v1beta1\\\",\\\"/healthz\\\",\\\"/healthz/etcd\\\",\\\"/healthz/log\\\",\\\"/healthz/ping\\\",\\\"/healthz/poststarthook/crd-informer-synced\\\",\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\",\\\"/healthz/poststarthook/start-apiextensions-controllers\\\",\\\"/healthz/poststarthook/start-apiextensions-informers\\\",\\\"/metrics\\\",\\\"/openapi/v2\\\",\\\"/version\\\"]}\",\"reason\":\"The descriptor url 'https://SOME_IP/swaggerapi/api/v1' for type provider 'mycluster-cluster-type' could not be fetched.\"}""}

[ingernet]

I am also seeing this error, with a codebase that was successful when we ran it in November.

my-fancy-cluster-type: {"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/my-fancy-cluster-type->$.descriptorUrl\n\"{\\\"originalResponse\\\":\\\"{\\\\\\\"paths\\\\\\\":[\\\\\\\"/apis\\\\\\\",\\\\\\\"/apis/\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\",\\\\\\\"/healthz\\\\\\\",\\\\\\\"/healthz/etcd\\\\\\\",\\\\\\\"/healthz/log\\\\\\\",\\\\\\\"/healthz/ping\\\\\\\",\\\\\\\"/healthz/poststarthook/crd-informer-synced\\\\\\\",\\\\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\\\\",\\\\\\\"/metrics\\\\\\\",\\\\\\\"/openapi/v2\\\\\\\",\\\\\\\"/version\\\\\\\"]}\\\",\\\"reason\\\":\\\"The descriptor url 'https://<IP-ADDRESS-REDACTED>/swaggerapi/api/v1' for type provider 'my-fancy-cluster-type' could not be fetched.\\\"}\""}
my-fancy-cluster-type-apps: {"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/my-fancy-cluster-type-apps->$.descriptorUrl\n\"{\\\"originalResponse\\\":\\\"{\\\\\\\"paths\\\\\\\":[\\\\\\\"/apis\\\\\\\",\\\\\\\"/apis/\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\",\\\\\\\"/healthz\\\\\\\",\\\\\\\"/healthz/etcd\\\\\\\",\\\\\\\"/healthz/log\\\\\\\",\\\\\\\"/healthz/ping\\\\\\\",\\\\\\\"/healthz/poststarthook/crd-informer-synced\\\\\\\",\\\\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\\\\",\\\\\\\"/metrics\\\\\\\",\\\\\\\"/openapi/v2\\\\\\\",\\\\\\\"/version\\\\\\\"]}\\\",\\\"reason\\\":\\\"The descriptor url 'https://<IP-ADDRESS-REDACTED>/swaggerapi/apis/apps/v1beta1' for type provider 'my-fancy-cluster-type-apps' could not be fetched.\\\"}\""}
my-fancy-cluster-type-v1beta1-extensions: {"ResourceType":"deploymentmanager.v2beta.typeProvider","ResourceErrorCode":"DESCRIPTOR_URL_FETCH_ERROR","ResourceErrorMessage":"location: /typeProviders/my-fancy-cluster-type-v1beta1-extensions->$.descriptorUrl\n\"{\\\"originalResponse\\\":\\\"{\\\\\\\"paths\\\\\\\":[\\\\\\\"/apis\\\\\\\",\\\\\\\"/apis/\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io\\\\\\\",\\\\\\\"/apis/apiextensions.k8s.io/v1beta1\\\\\\\",\\\\\\\"/healthz\\\\\\\",\\\\\\\"/healthz/etcd\\\\\\\",\\\\\\\"/healthz/log\\\\\\\",\\\\\\\"/healthz/ping\\\\\\\",\\\\\\\"/healthz/poststarthook/crd-informer-synced\\\\\\\",\\\\\\\"/healthz/poststarthook/generic-apiserver-start-informers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-controllers\\\\\\\",\\\\\\\"/healthz/poststarthook/start-apiextensions-informers\\\\\\\",\\\\\\\"/metrics\\\\\\\",\\\\\\\"/openapi/v2\\\\\\\",\\\\\\\"/version\\\\\\\"]}\\\",\\\"reason\\\":\\\"The descriptor url 'https://<IP-ADDRESS-REDACTED>/swaggerapi/apis/extensions/v1beta1' for type provider 'my-fancy-cluster-type-v1beta1-extensions' could not be fetched.\\\"}\""}

[ANT1OFF]

I believe the reason is that GKE now defaults to version 1.14 where the swagger endpoint has been removed and replaced with the openapi/v2 endpoint. As a result, all custom types that use that endpoint fails.
The error message is the default response from the API server when an unknown endpoint is accessed.

[mmosca]

The missing swagger endpoint can be fixed by replacing /swaggerapi/ with /openapi/v2/,

Line 97 in cluster.py should look like this:
'https://$(ref.', cluster_name, '.endpoint)/openapi/v2/',

I am not sure if that is enough to fix the issue, as I am getting some errors while trying to create namespaces on my modified script.

[rjg21]

Swapping the endpoint to /openapi/v2 removes the descriptor could not be fetched error but I now get this:

- code: MISSING_METHOD_IN_COLLECTION
   message: Method 'insert' does not exist for collection '/apis/batch/v1beta1/namespaces/{namespace}/cronjobs'
     in descriptor url 'https://<REDACTED>/openapi/v2'

[mmosca]

With some help from Google Engineers I managed to get my deployment working.

You no longer need multiple type providers, this should be enough:

resources.append({
'name': type_name,
'type': 'deploymentmanager.v2beta.typeProvider',
'properties': {
'options': {
'validationOptions': {
# Kubernetes API accepts ints, in fields they annotate
# with string. This validation will show as warning
# rather than failure for Deployment Manager.
# kubernetes/kubernetes#2971
'schemaValidation': 'FAIL'
},
# According to kubernetes spec, the path parameter 'name' and 'namespace'
# must be set inside the metadata field.
# https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#metadata
# The input mappings will map the values to path parameters
'inputMappings': [{
'fieldName': 'name',
'location': 'PATH',
'methodMatch': '^(get|delete|put|patch|post)$',
'value': '$.resource.properties.metadata.name'
}, {
'fieldName': 'namespace',
'location': 'PATH',
'methodMatch': '^(get|delete|put|patch|post)$',
'value': '$.resource.properties.metadata.namespace'
}, {
'fieldName': 'metadata.resourceVersion',
'location': 'BODY',
'methodMatch': '^(put|patch)$',
'value': '$.resource.self.metadata.resourceVersion'
}, {
'fieldName': 'Authorization',
'location': 'HEADER',
'value': '$.concat("Bearer ",'
'$.googleOauth2AccessToken())'
}]
},
'collectionOverrides': [{
'collection': '/api/v1/namespaces/{namespace}/services/{name}',
'options': {
'inputMappings': [{
'fieldName': 'spec.clusterIP',
'location': 'BODY',
'methodMatch': '^(put|patch)$',
'value': '$.resource.self.spec.clusterIP'
}]
}
}],
'descriptorUrl': 'https://$(ref.' + cluster_name + '.endpoint)/openapi/v2'
}
})

You have to remove extra qualifiers from your cluster_types and add {name} to the end of the url.

Something like this:

cluster_types = {
'Service': ''.join([
cluster_types_root,
':',
'/api/v1/namespaces/{namespace}/services/{name}'
]),
'Deployment': ''.join([
cluster_types_root,
':',
'/apis/apps/v1beta1/namespaces/{namespace}/deployments/{name}'
]),
...

[ingernet]

@mmosca, thanks for sharing your code! with a couple of tweaks, i got it working. here's my project fork if anybody needs a hand: https://github.com/ingernet/deploymentmanager-samples

[korutx]

@mmosca @ingernet thanks for sharing. I recently had the same problem, but the @ingernet code no longer works, because the Deployment Resource API changed once again. I will provide utility commands for diagnosis:

1. Get descriptor URL

DESCRIPTOR_URL=`gcloud beta deployment-manager type-providers describe ${CLUSTER_NAME}-cluster-py-type --format='value(descriptorUrl)'`

2. Download the current swagger definition

curl -sk -H "Authorization: Bearer `gcloud auth print-access-token`" $DESCRIPTOR_URL | jq > swagger_default.json 

4. Find the correct resource endpoint and adjust your references
   In my case this is the proper endpoint /apis/apps/v1/namespaces/{namespace}/deployments/{name}

I found this article really useful: https://medium.com/google-cloud/managing-kubernetes-customresourcedefinitions-with-google-deployment-manager-b9feeadae126

Hope it helps!