HMR port config not respected

[bailout00]

Describe the bug

server: {
    port: 4000,      
    hmr: { port: 3001 },
    hmr: { port: 3001, clientPort: 3001 },

And several other config combinations did not work, (Unable to connect to websocket in Chrome/FF). However, using:

server: {
    port: 4000,    
    hmr: { port: 4000 },

Works fine. I'm assuming somewhere that port is not being respected. This occurred immediately following an upgrade from 4.1.1 to 4.3.1

Reproduction

https://stackblitz.com/edit/vitejs-vite-ked8jv?file=vite.config.ts,main.js&terminal=dev

Steps to reproduce

Add

server: 4000,
hmr: {  port: 3001 },

to a bare-bones config and WS will fail to connect.

change to port to clientPort and match server port, and it will work.

Notably, netstat confirms that no matter what port is in hmr.port it is not listening on that port

% netstat -ant | grep 3001 | grep LISTEN
% 
% netstat -ant | grep 4000 | grep LISTEN
tcp4       0      0  127.0.0.1.4000         *.*                    LISTEN     
% 

Was able to reproduce in 4.3.0 on repro above. It actually sent the browser into a frenzy trying to reconnect.

System Info

System:
    OS: macOS 12.6.5
    CPU: (10) arm64 Apple M1 Max
    Memory: 15.97 GB / 64.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.16.0 - ~/.nvm/versions/node/v18.16.0/bin/node
    Yarn: 1.22.19 - /opt/homebrew/bin/yarn
    npm: 9.5.1 - ~/.nvm/versions/node/v18.16.0/bin/npm
    Watchman: 2023.03.06.00 - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 112.0.5615.137
    Edge: 112.0.1722.58
    Firefox: 112.0.1
    Safari: 16.4.1
  npmPackages:
    @vitejs/plugin-react-swc: ^3.3.0 => 3.3.0 
    vite: ^4.3.1 => 4.3.1 

### Used Package Manager

npm

### Logs

<details>
<summary>Click to expand!</summary>

WebSocket connection to 'wss://localhost:3001/' failed:
client.ts:102[vite] server connection lost. polling for restart...
https:/…4000/@vite/client:445GET https://localhost:3001/ net::ERR_CONNECTION_REFUSED

### Validations

- [X] Follow our [Code of Conduct](https://github.com/vitejs/vite/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitejs/vite/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitejs.dev/guide).
- [X] Check that there isn't [already an issue](https://github.com/vitejs/vite/issues) that reports the same bug to avoid creating a duplicate.
- [X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to [vuejs/core](https://github.com/vuejs/core) instead.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitejs/vite/discussions) or join our [Discord Chat Server](https://chat.vitejs.dev/).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

[stackblitz]

Start a new pull request in StackBlitz Codeflow.

• vitejs-vite-ked8jv (Open reproduction)

[sapphi-red]

It seems this happens since 4.3.0-beta.2.

[patak-dev]

Maybe related to this PR #12734? cc @sun0day

[sapphi-red]

Tried reverting #12734 locally and it worked. It seems that PR caused this one.

[sun0day]

Maybe related to this PR #12734? cc @sun0day

Located the bug. WIP