Identify which node a response comes from

[john-thomas-dotcms]

Is your feature request related to a problem? Please describe.

There are many types of support issues where identifying which node responded to a request is critical to resolving the issue (to identify a node which isn't responding correctly, identify out-of-sync nodes, etc.). Currently there is no standard way to do this; it's been done by allowing access to each node in the cluster via separate URLs, modifying load balancer configuration, and other methods, but none of these methods works well, and none of them work universally for all systems.

This all means that each time this information is needed, a custom solution has to be implemented (and then usually rolled back after the problem has been solved).

Describe the solution you'd like

We need a way for dotCMS to identify which node in a cluster each specific response is sent from. This is what I suggest, but there are multiple ways this could be implemented:

• Add a response header which identifies which node the response was sent from.
• Enable or disable the response header to all responses via a configuration property (for example, RESPONSE_HEADER_ADD_NODE_ID=true).
• Allow enabling the response header in an individual request by adding a URL parameter (for example, ?nodeid=true).

Describe alternatives you've considered

dotCMS Cloud Engineering has considered implementing something of this nature on the load balancer for all dotCMS Cloud customers. However, this isn't a workable option within our current infrastructure (ALB), and there are undoubtedly on-premise customers with similar limitations.

Other methods, such as bypassing the LB to access nodes directly, present performance and reliability issues, so they can't be implemented for any customers on more than a temporary (custom) basis.

Additional context

The identifier used in the header doesn't matter, as long as:

• It's unique to each node.
• It can be easily used to identify the node within the infrastructure.

Ideally, the node identifier should change if the server or container running on the node is changed (especially with k8s, where multiple containers may be started from the same image).

• But if necessary, a unique ID (for the server/image) and a timestamp (for when the server/container was started) could potentially be both used to ensure identification of unique containers.

The Site from the license screen seems to fit most of these needs, so that might be an appropriate identifier to use. If the Site doesn't change when k8s shuts down and restarts a container, then it might make sense to send a timestamp (or maybe Server ID) in addition to the Site.

[jcastro-dotcms]

One of the customers requesting this feature: https://dotcms.zendesk.com/agent/tickets/105784

[wezell]

We should send a x-dot-server header that also includes the shorty serverId. This can be disabled by passing the environmental variable RESPONSE_HEADER_ADD_NODE_ID=false

[wezell]

So we should send 2 pieces of information, the shorty serverId and the node name, which is found here:

• The inclusion of the node name should be disable-able via config
• We need to make sure that the node name is properly cached.

The header should look something like:
{nodeName} | {serverId}

x-dot-server: dotcms-node-1 | dde63dc7

[jdotcms]

Needs doc:
RESPONSE_HEADER_ADD_NODE_ID=false

This one will avoids the inclusion of the header at all on the http response

RESPONSE_HEADER_ADD_NODE_ID_INCLUDE_NODE_NAME=false

This one will avoids the inclusion of the node name into the x-dot-server header, in summary it will returns on the response:

x-dot-server: unknown|{serverId}

[wezell]

So, to be clear
RESPONSE_HEADER_ADD_NODE_ID=false means no header will be added and
RESPONSE_HEADER_ADD_NODE_ID_INCLUDE_NODE_NAME=false means it will only show the nodeId and not Node Name.

[fmontes]

[bryanboza]

Fixed, tested on release-22.02 // Docker // FF

By default those are the results:

After turn of the follow properties:

RESPONSE_HEADER_ADD_NODE_ID=false
RESPONSE_HEADER_ADD_NODE_ID_INCLUDE_NODE_NAME=false

We are not including the header