Build Clusters with Nodes which are behind LB / NAT

[vccddd]

I'm having some problems trying to build the cluster.

Some of the nodes in this cluster are deployed in Docker and they don't have a public IP that they can bind to, I looked at the documentation and used --cluster-announce-ip to declare their public IP, but in the code I noticed a judgment call that this declared IP must be present in the --bind.

garnet/libs/host/Configuration/Options.cs

Lines 719 to 725 in 297d0a3

	if (ClusterAnnounceIp != null)
	{
	ClusterAnnouncePort = ClusterAnnouncePort == 0 ? Port : ClusterAnnouncePort;
	clusterAnnounceEndpoint = Format.TryCreateEndpoint(ClusterAnnounceIp, ClusterAnnouncePort, tryConnect: false, logger: logger).GetAwaiter().GetResult();
	if (clusterAnnounceEndpoint == null || !endpoints.Any(endpoint => endpoint.Equals(clusterAnnounceEndpoint[0])))
	throw new GarnetException("Cluster announce endpoint does not match list of listen endpoints provided!");
	}

All configuration here is passed through the CLI. Since Garnet is not compatible with all configurations in redis.conf, it's possible that the meaning of the parameters here may not be the same as in Redis.

I'd like to ask if there's any way to go about building such a cluster.

[vazois]

The endpoint the server listens to is defined using --bind and --port. You can use multiple addresses but only a single port.
When you provide that information the instance creates sockets listening to these endpoints.
For this reason, the --cluster-announce-ip and --cluster-announce-port has to match one of these endpoints, because otherwise how will the nodes communicate with each other.

The only improvement that I can see is make it so that --cluster-announce-ip and --cluster-announce-port creates an additional socket to which the server will be listening too.
But this just expands, the list of endpoints we need to create irrespective of if they are public or private.
So, I am not sure if it will make a difference in what you are trying to do.
If you want a specific ip for cluster why don't you start the server like:

--bind="ipAddress" --port="port" --cluster-announce-ip="ipAddress" --cluster-announce-port="port"

This creates a socket using the endpoint (ipAddress,port) and tells the cluster to listen to that endpoint.

Alternatively, for docker you can leave the --bind and --cluster-announce options empty and the server listens to all ips available and the cluster should be able to figure out the ipAddress to advertise.

[vccddd]

Let me give you an example. This is a Redis replica cluster.

Simply put, there is a configuration section in the Redis documentation that represents all the issues.
https://redis.io/docs/latest/operate/oss_and_stack/management/replication/#configuring-replication-in-docker-and-nat

I admit I'm setting up an extremely unprofessional cluster, but for a simple application, this is a relatively quick solution to achieve cross-region data synchronization.

I have two machines here:

master1

HOST_MACHINE[1.2.3.4] -> DOCKER_CONTAINER[172.18.0.2]

slave1

LOAD_BALANCER[1.2.3.5] -> HOST_MACHINE[192.168.0.10] -> DOCKER_CONTAINER[172.18.0.2]

This raises a problem.

1. On master1, the container's IP is 172.18.0.2. To run it and make it recognizable by other machines, it must be bound to [172.18.0.2, 1.2.3.4]. However, the issue is that this container cannot bind to the host machine's network interface with the IP address 1.2.3.4, resulting in the error: “Unable to initialize server due to exception: Cannot assign requested address.”

2. On slave1, the network interface's IP address is not a public IP address, so it faces the same issue as master1. Its IP address is 192.168.12.10, which cannot listen on the LB's IP address 1.2.3.5

If I leave both parameters blank, starting and binding the cluster will work without any issues, but data synchronization will not occur (the cluster nodes display both IP addresses as container IP addresses).

[vazois]

Note that the link you shared refers to sentinel which is currently not supported.
Also, in you example you are binding both nodes to the same port. Maybe another solution could be to specify different port for every instance.
That being said, it might be as simple as adding a parameter in Garnet such as --cluster-endpoint-skip-validate.
This will offload endpoint validation to the operator of the cluster.
We don't have a plan for this but if you want to contribute to it, it will be great.