Google BigQuery support

[gabriel-milan]

Hello there!

Do you consider supporting BigQuery? If so, could you give an idea of how to add support for it?

[KentonParton]

Hi @gabriel-milan. Yes, we'd love to support BigQuery. I can get it added this weekend if that works?

[gabriel-milan]

Hi! I'm sorry about the delay.

The swiple/swiple-ui:pr-67 image is raising the following error:

/docker-entrypoint.sh: exec: line 47: npm: not found

Apart from that, is there any progress about either cloud support or a Helm chart?

[KentonParton]

Hi @gabriel-milan! Thanks for giving it a try and sorry about the error. I will look into this today.

Regarding cloud support, we expect Swiple Cloud to be available to customers in 1 - 2 months. Customers that join early will have access to a managed Swiple environment at no cost for the first 12 months. Is this something your company is interested in?

[gabriel-milan]

Is this something your company is interested in?

We'd be glad to try that out! But, to be honest, a Helm chart would fit us better, since we'd rather host Swiple in our infrastructure. Is this something you'd be interested in making publicly available?

[KentonParton]

Apologies @gabriel-milan, could you make the following changes to docker-compose-non-dev.yml instead of docker-compose.yml:

1. Update swiple/swiple-api:main to swiple/swiple-api:pr-67
2. Update swiple/swiple-ui:main to swiple/swiple-ui:pr-67

Then run:
docker compose -f docker-compose-non-dev.yaml up

This should run as expected.

[gabriel-milan]

Unfortunately, I've had another issue:

setup                  | {'user': {'index_name': 'user', 'mappings': {'properties': {'oauth_accounts': {'type': 'nested'}}}}, 'datasoures': {'index_name': 'datasources', 'mappings': {'properties': {'account_name': {'type': 'keyword'}, 'create_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}, 'created_by': {'type': 'keyword'}, 'database': {'type': 'keyword'}, 'dataset': {'type': 'keyword'}, 'datasource_name': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'description': {'type': 'text'}, 'engine': {'type': 'keyword'}, 'gcp_project': {'type': 'keyword'}, 'host': {'type': 'keyword'}, 'modified_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}, 'password': {'type': 'keyword'}, 'port': {'type': 'integer'}, 'region': {'type': 'keyword'}, 'role_name': {'type': 'keyword'}, 's3_staging_dir': {'type': 'keyword'}, 'schema_name': {'type': 'keyword'}, 'ssl_mode': {'type': 'boolean'}, 'username': {'type': 'keyword'}, 'warehouse': {'type': 'keyword'}}}}, 'datasets': {'index_name': 'datasets', 'mappings': {'properties': {'create_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}, 'created_by': {'type': 'keyword'}, 'data_asset_name': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'dataset_name': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'datasource_id': {'type': 'keyword'}, 'engine': {'type': 'keyword'}, 'modified_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}, 'runtime_parameters': {'properties': {'query': {'type': 'keyword'}, 'schema': {'type': 'keyword'}}, 'type': 'object'}}}}, 'expectations': {'index_name': 'expectations', 'mappings': {'properties': {'create_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}, 'dataset_id': {'type': 'keyword'}, 'datasource_id': {'type': 'keyword'}, 'expectation_type': {'type': 'keyword'}, 'kwargs': {'type': 'text'}, 'modified_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}}}}, 'validations': {'index_name': 'validations', 'mappings': {'properties': {'meta': {'type': 'object', 'properties': {'run_id': {'type': 'object', 'properties': {'run_time': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}}}}}}}}, 'actions': {'index_name': 'actions', 'mappings': {'properties': {'resource_key': {'type': 'keyword'}, 'resource_type': {'type': 'keyword'}, 'action_type': {'type': 'keyword'}, 'destination': {'type': 'object', 'properties': {'key': {'type': 'keyword'}, 'destination_name': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'destination_type': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'kwargs': {'type': 'object'}, 'create_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}, 'created_by': {'type': 'keyword'}, 'modified_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}}}}}}, 'destinations': {'index_name': 'destinations', 'mappings': {'properties': {'destination_name': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'destination_type': {'fielddata': True, 'fields': {'keyword': {'type': 'keyword'}}, 'type': 'text'}, 'kwargs': {'type': 'object'}, 'create_date': {'format': 'yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ', 'type': 'date'}}}}}
setup                  | Traceback (most recent call last):
setup                  |   File "/code/app/scripts/setup_opensearch.py", line 27, in <module>
setup                  |     create_indicies(client)
setup                  |   File "/code/app/scripts/setup_opensearch.py", line 15, in create_indicies
setup                  |     response = client.indices.create(
setup                  |   File "/code/.venv/lib/python3.9/site-packages/opensearchpy/client/utils.py", line 177, in _wrapped
setup                  |     return func(*args, params=params, headers=headers, **kwargs)
setup                  |   File "/code/.venv/lib/python3.9/site-packages/opensearchpy/client/indices.py", line 128, in create
setup                  |     return self.transport.perform_request(
setup                  |   File "/code/.venv/lib/python3.9/site-packages/opensearchpy/transport.py", line 410, in perform_request
setup                  |     raise e
setup                  |   File "/code/.venv/lib/python3.9/site-packages/opensearchpy/transport.py", line 373, in perform_request
setup                  |     status, headers_response, data = connection.perform_request(
setup                  |   File "/code/.venv/lib/python3.9/site-packages/opensearchpy/connection/http_urllib3.py", line 282, in perform_request
setup                  |     self._raise_error(response.status, raw_data)
setup                  |   File "/code/.venv/lib/python3.9/site-packages/opensearchpy/connection/base.py", line 330, in _raise_error
setup                  |     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
setup                  | opensearchpy.exceptions.TransportError: TransportError(503, 'OpenSearch Security not initialized.')
setup exited with code 1