Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(codec) Debounce the call that calc codec intersection set. #2622

Merged
merged 2 commits into from
Jan 27, 2025
Merged

fix(codec) Debounce the call that calc codec intersection set. #2622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7042a5a
fix(codec) Debounce the call that calc codec intersection set.
jallamsetty1 Jan 24, 2025
b35ec07
squash: Address review comments
jallamsetty1 Jan 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions modules/RTC/TPCUtils.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,10 +512,10 @@ export class TPCUtils {
const rtpSender = this.pc.findSenderForTrack(localVideoTrack.getTrack());

if (this.pc.usesCodecSelectionAPI() && rtpSender) {
const { codecs } = rtpSender.getParameters();
const { encodings } = rtpSender.getParameters();
saghul marked this conversation as resolved.
Show resolved Hide resolved

if (codecs?.length) {
return codecs[0].mimeType.split('/')[1].toLowerCase();
if (encodings[0].codec) {
return encodings[0].codec.mimeType.split('/')[1].toLowerCase();
}
}

Expand Down
26 changes: 18 additions & 8 deletions modules/RTC/TraceablePeerConnection.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1436,8 +1436,18 @@ TraceablePeerConnection.prototype.setVideoCodecs = function(codecList, screensha
this.codecSettings.screenshareCodec = screenshareCodec;
}

if (this.usesCodecSelectionAPI()) {
this.configureVideoSenderEncodings();
if (!this.usesCodecSelectionAPI()) {
return;
}

for (const track of this.getLocalVideoTracks()) {
const currentCodec = this.tpcUtils.getConfiguredVideoCodec(track);

if (screenshareCodec && track.getVideoType() === VideoType.DESKTOP && screenshareCodec !== currentCodec) {
this.configureVideoSenderEncodings(track, screenshareCodec);
} else if (currentCodec !== codecList[0]) {
this.configureVideoSenderEncodings(track);
}
}
};

Expand Down Expand Up @@ -1904,16 +1914,16 @@ TraceablePeerConnection.prototype._enableSenderEncodings = async function(sender
* that is currently selected.
*
* @param {JitsiLocalTrack} - The local track for which the sender encodings have to configured.
* @param {CodecMimeType} - The preferred codec for the video track.
* @returns {Promise} promise that will be resolved when the operation is successful and rejected otherwise.
*/
TraceablePeerConnection.prototype.configureVideoSenderEncodings = function(localVideoTrack = null) {
const preferredCodec = this.codecSettings.codecList[0];
TraceablePeerConnection.prototype.configureVideoSenderEncodings = function(localVideoTrack, codec) {
const preferredCodec = codec ?? this.codecSettings.codecList[0];

if (localVideoTrack) {
return this.setSenderVideoConstraints(
this._senderMaxHeights.get(localVideoTrack.getSourceName()),
localVideoTrack,
preferredCodec);
const height = this._senderMaxHeights.get(localVideoTrack.getSourceName()) ?? VIDEO_QUALITY_LEVELS[0].height;

return this.setSenderVideoConstraints(height, localVideoTrack, preferredCodec);
}
const promises = [];

Expand Down
113 changes: 93 additions & 20 deletions modules/qualitycontrol/CodecSelection.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,43 +50,57 @@ describe('Codec Selection', () => {
};

qualityController = new QualityController(conference, options);
jasmine.clock().install();
spyOn(jingleSession, 'setVideoCodecs');
});

it('and remote endpoints use the new codec selection logic', () => {
afterEach(() => {
jasmine.clock().uninstall();
});

it('and remote endpoints use the new codec selection logic', async () => {
// Add a second user joining the call.
participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'vp9', 'vp8' ]);

await nextTick(1000);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp9', 'vp8' ], 'vp9');

// Add a third user joining the call with a subset of codecs.
participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'vp8' ]);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], 'vp9');

// Make p2 leave the call
// Make p2 leave the call.
conference.removeParticipant(participant2);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(3);

await nextTick(1000);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(2);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp9', 'vp8' ], 'vp9');
});

it('and remote endpoints use the old codec selection logic (RN)', () => {
it('and remote endpoints use the old codec selection logic (RN)', async () => {
// Add a second user joining the call.
participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, null, 'vp8');

await nextTick(1000);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], 'vp9');

// Add a third user (newer) to the call.
participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'vp9', 'vp8' ]);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], 'vp9');

// Make p1 leave the call
conference.removeParticipant(participant1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(3);

await nextTick(1000);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(2);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp9', 'vp8' ], 'vp9');
});
});

Expand All @@ -102,50 +116,61 @@ describe('Codec Selection', () => {

qualityController = new QualityController(conference, options);
spyOn(jingleSession, 'setVideoCodecs');
jasmine.clock().install();
});

it('and remote endpoints use the new codec selection logic', () => {
afterEach(() => {
jasmine.clock().uninstall();
});

it('and remote endpoints use the new codec selection logic', async () => {
// Add a second user joining the call.
participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'vp9', 'vp8', 'h264' ]);

await nextTick(1000);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp9', 'vp8' ], undefined);

// Add a third user joining the call with a subset of codecs.
participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'vp8' ]);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], undefined);

// Make p2 leave the call
conference.removeParticipant(participant2);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(3);

await nextTick(1000);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(2);
});

it('and remote endpoint prefers a codec that is locally disabled', () => {
it('and remote endpoint prefers a codec that is locally disabled', async () => {
// Add a second user joining the call the prefers H.264 and VP8.
participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'h264', 'vp8' ]);

await nextTick(1200);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], undefined);
});

it('and remote endpoints use the old codec selection logic (RN)', () => {
it('and remote endpoints use the old codec selection logic (RN)', async () => {
// Add a second user joining the call.
participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, null, 'vp8');

await nextTick(1000);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], undefined);

// Add a third user (newer) to the call.
participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'vp9', 'vp8', 'h264' ]);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8' ], undefined);

// Make p1 leave the call
conference.removeParticipant(participant1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(3);

jasmine.clock().tick(1000);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(2);
});
});

Expand All @@ -171,11 +196,14 @@ describe('Codec Selection', () => {

participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'av1', 'vp9', 'vp8' ]);

await nextTick(1000);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'av1', 'vp9', 'vp8' ], undefined);

participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'av1', 'vp9', 'vp8' ]);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'av1', 'vp9', 'vp8' ], undefined);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);

qualityController.codecController.changeCodecPreferenceOrder(localTrack, 'av1');

Expand All @@ -187,6 +215,7 @@ describe('Codec Selection', () => {

// Expect the local endpoint to continue sending VP9.
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp9', 'av1', 'vp8' ], undefined);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(3);
});

it('and does not change codec if the current codec is already the lowest complexity codec', async () => {
Expand All @@ -196,6 +225,8 @@ describe('Codec Selection', () => {

participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'av1', 'vp9', 'vp8' ]);

await nextTick(1000);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp8', 'vp9', 'av1' ], undefined);

participant2 = new MockParticipant('remote-2');
Expand Down Expand Up @@ -239,11 +270,14 @@ describe('Codec Selection', () => {

participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'av1', 'vp9', 'vp8' ]);

await nextTick(1000);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'av1', 'vp9', 'vp8' ], undefined);

participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'av1', 'vp9', 'vp8' ]);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'av1', 'vp9', 'vp8' ], undefined);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
saghul marked this conversation as resolved.
Show resolved Hide resolved

const sourceStats = {
avgEncodeTime: 12,
Expand All @@ -269,6 +303,8 @@ describe('Codec Selection', () => {
participant3 = new MockParticipant('remote-3');
conference.addParticipant(participant3, [ 'av1', 'vp9', 'vp8' ]);

await nextTick(1000);

// Expect the local endpoint to continue sending VP9.
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'vp9', 'av1', 'vp8' ], undefined);

Expand Down Expand Up @@ -328,4 +364,41 @@ describe('Codec Selection', () => {
expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(0);
});
});

describe('When multiple joins and leaves happen in a quick burst', () => {
beforeEach(() => {
options = {
jvb: {
preferenceOrder: [ 'AV1', 'VP9', 'VP8' ],
screenshareCodec: 'VP9'
},
p2p: {}
};
jasmine.clock().install();
qualityController = new QualityController(conference, options);
spyOn(jingleSession, 'setVideoCodecs');
});

afterEach(() => {
jasmine.clock().uninstall();
});

it('should call setVideoCodecs only once within the same tick', async () => {
participant1 = new MockParticipant('remote-1');
conference.addParticipant(participant1, [ 'vp9', 'vp8' ]);

// Add a third user joining the call with a subset of codecs.
participant2 = new MockParticipant('remote-2');
conference.addParticipant(participant2, [ 'vp8' ]);

// Make p1 and p2 leave the call.
conference.removeParticipant(participant2);
conference.removeParticipant(participant1);

await nextTick(1000);

expect(jingleSession.setVideoCodecs).toHaveBeenCalledTimes(1);
expect(jingleSession.setVideoCodecs).toHaveBeenCalledWith([ 'av1', 'vp9', 'vp8' ], 'vp9');
});
});
});
4 changes: 2 additions & 2 deletions modules/qualitycontrol/QualityController.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ describe('QualityController', () => {
p2p: {}
};
localTrack = new MockLocalTrack('1', 720, 'camera');
qualityController = new QualityController(conference, options, true);
qualityController = new QualityController(conference, options);
sourceStats = {
avgEncodeTime: 12,
codec: 'VP8',
Expand Down Expand Up @@ -165,7 +165,7 @@ describe('QualityController', () => {
p2p: {}
};
localTrack = new MockLocalTrack('1', 720, 'camera');
qualityController = new QualityController(conference, options, true);
qualityController = new QualityController(conference, options);
sourceStats = {
avgEncodeTime: 12,
codec: 'VP8',
Expand Down
33 changes: 27 additions & 6 deletions modules/qualitycontrol/QualityController.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,12 +145,13 @@ export class QualityController {
this._conference.on(
JitsiConferenceEvents.CONFERENCE_VISITOR_CODECS_CHANGED,
(codecList: CodecMimeType[]) => this._codecController.updateVisitorCodecs(codecList));
this._conference.on(
JitsiConferenceEvents.USER_JOINED,
() => this._codecController.selectPreferredCodec(this._conference.jvbJingleSession));
this._conference.on(
JitsiConferenceEvents.USER_LEFT,
() => this._codecController.selectPreferredCodec(this._conference.jvbJingleSession));

// Debounce the calls to codec selection when there is a burst of joins and leaves.
const debouncedSelectCodec = this._debounce(
() => this._codecController.selectPreferredCodec(this._conference.jvbJingleSession),
1000);
this._conference.on(JitsiConferenceEvents.USER_JOINED, debouncedSelectCodec.bind(this));
this._conference.on(JitsiConferenceEvents.USER_LEFT, debouncedSelectCodec.bind(this));
this._conference.rtc.on(
RTCEvents.SENDER_VIDEO_CONSTRAINTS_CHANGED,
(videoConstraints: IVideoConstraints) => this._sendVideoController.onSenderConstraintsReceived(videoConstraints));
Expand All @@ -159,6 +160,26 @@ export class QualityController {
(tpc: TraceablePeerConnection, stats: Map<number, IOutboundRtpStats>) => this._processOutboundRtpStats(tpc, stats));
}

/**
* Creates a debounced function that delays the execution of the provided function until after the specified delay
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this basically lodash's throtlle? https://lodash.com/docs/4.17.15#throttle

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, I see you imported it but then didn't use it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, I was testing with lodash throttle with both the leading and trailing options set to true and ran into this limitation - If leading and trailing options are true, func is invoked on the trailing edge of the timeout only if the throttled function is invoked more than once during the wait timeout.
I can go back and test lodash throttle with only the trailing option set to true and I believe the end result will be the same. Will update this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Member Author

@jallamsetty1 jallamsetty1 Jan 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually when I pass { leading: false, trailing: true } to lodash's throttle, my testcases fail, its missing some of the calls when we expect them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are waiting for 1000 and advancing by 1000 I bet there are shenanigans there? Perhaps we should test by waiting a bit longer?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the same results when I had the timer set to 1200 ms. I remember this being the case when we initially discussed about debouncing this call and my unit tests for testing lodash's throttle had also failed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok let's have it this way 👍

* has elapsed. Unlike typical debounce implementations, the timer does not reset when the function is called again
* within the delay period.
*
* @param {Function} func - The function to be debounced.
* @param {number} delay - The delay in milliseconds.
* @returns {Function} - The debounced function.
*/
_debounce(func: Function, delay: number) {
return function (...args) {
if (!this._timer) {
this._timer = setTimeout(() => {
this._timer = null;
saghul marked this conversation as resolved.
Show resolved Hide resolved
func.apply(this, args);
}, delay);
}
};
}

/**
* Adjusts the lastN value so that fewer remote video sources are received from the bridge in an attempt to improve
* encode resolution of the outbound video streams based on cpuLimited parameter passed. If cpuLimited is false,
Expand Down
3 changes: 2 additions & 1 deletion modules/xmpp/JingleSessionPC.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2283,7 +2283,6 @@ export default class JingleSessionPC extends JingleSession {
*/
setVideoCodecs(codecList, screenshareCodec) {
if (this._assertNotEnded()) {
logger.info(`${this} setVideoCodecs: codecList=${codecList}, screenshareCodec=${screenshareCodec}`);
this.peerconnection.setVideoCodecs(codecList, screenshareCodec);

// Browser throws an error when H.264 is set on the encodings. Therefore, munge the SDP when H.264 needs to
Expand All @@ -2307,6 +2306,8 @@ export default class JingleSessionPC extends JingleSession {
videoType: VideoType.CAMERA
});

logger.info(`${this} setVideoCodecs: codecList=${codecList}, screenshareCodec=${screenshareCodec}`);

// Initiate a renegotiate for the codec setting to take effect.
const workFunction = finishedCallback => {
this._renegotiate()
Expand Down
Loading