[webm-discuss] Issue 1580 in webm: vp9 encoder in SVC flexible mode and skipped layers can create strange spatial references

[il… via monorail]

Status: Unconfirmed
Owner: ----

New issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible mode and
skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580

We are using vp9 flexible svc mode in webrtc with different fps per spatial
layer. Idea is that vp9 will provide the following references (no temporal
layers involved):

o---o---o---o---o
| |
o---------------o
| |
o---------------o
(o - means subframe, horizontal or vertical line is reference. Bottom is
the base layer, 3 spatial layers are used).

Encoder is asked to skip some layers by calling:
vpx_codec_control(encoder_, VP9E_SET_SVC_LAYER_ID, &layer_id);

Where layer_id.spatial_layer_id > 0.

What happens in fact is that all frames with only 2nd spatial layer also
have a reference to the last encoded 1st spatial layer subframe.


So, this is what actually happening:

o---o---o
| / /
o------
|
o


The Last buffer references previous frame on the same layer correctly, but
golden buffer contains a strange reference to the previous layer of the
previous picture.

We have a workaround for that in WebRTC, but it may be an unintended
behavior, please check if it's a bug.
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[il… via monorail]

Comment #1 on issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c1

Somehow, I can't edit the issue.

Could someone please add ***@chromium.org to cc?
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[il… via monorail]

Comment #2 on issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c2

It looks like encoder doesn't check what is actually in the buffer, when it
references it.

The more strange thing happens if there's an upswitch at a high-layer only
frame:


o
/
o--o
| |
o--o

The last frame has only SL 2. The frist frames have only SLs 0 and 1. The
last frame somehow is referencing both 1:1 and 0:0, which is very weird.
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[johannko… via monorail]

Updates:
Cc: ***@chromium.org

Comment #3 on issue 1580 by ***@google.com: vp9 encoder in SVC
flexible mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c3

(No comment was entered for this change.)
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[il… via monorail]

Comment #4 on issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c4

I think this may actually solve the issue already:
https://bugs.chromium.org/p/webm/issues/detail?id=1526
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[il… via monorail]

Comment #5 on issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c5

The INTER_LAYER_PRED_ON_CONSTRAINED INTER_LAYER_PRED mode would solve our
problems (vp9 encoder generates non RTP compatible bitstream --- all
reference to different superframes must be exactly to the same spatial
layer).

The problem now is that it doesn't work. I confirmed that I set
INTER_LAYER_PRED to 3 instead of 0, and the same references are reported by
the encoder (queried using VP9E_GET_SVC_REF_FRAME_CONFIG).

Even if encoder doesn't actually use these references we can't check that
in webrtc.


To reproduce this,
1) set INTER_LAYER_PRED to 3 and encode 2 frames with explicitly disabled
2nd spatial layer (bitrate=0)
2) set the bitrate for last layer to something non-negative
3) encode only last layer by setting VP9E_SET_SVC_LAYER_ID to 2 and encode
the 3rd frame.

Expected result is that the frame would have no references, but it would in
fact reference 0th spatial layer of the very first frame and 1st spatial
layer of the second frame.


Even if the last frame were encoded starting from spatial layer 1, the 2nd
spatial layer would have a bogus dependency on frame 0, spatial layer 0.

My understanding is that encoder uses the buffer, which would have a 2nd
spatial layer in a previous frame, as a temporal reference. But since the
frame was not encoded, the buffer contains something else (0th spatial
layer of 0th frame).

Please add some checks for what exactly buffers contain and only use
buffers if spatial_layer_id or picture_num are the same as a current frame
in INTER_LAYER_PRED_ON_CONSTRAINED mode.
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[il… via monorail]

Comment #6 on issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c6

Please add ***@chromium.org to cc.

Also could you please bump a priority of this issue a little? We would like
to test some high fps vp9 screenshare in M72, and this issue is blocking
this. I would appreciate if you manage to merge the fix to M72 branch.

I am very sorry for reporting this so late.
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[johannko… via monorail]

Updates:
Cc: ***@google.com ***@google.com ***@chromium.org

Comment #7 on issue 1580 by ***@google.com: vp9 encoder in SVC
flexible mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c7

(No comment was entered for this change.)
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[il… via monorail]

Comment #8 on issue 1580 by ***@chromium.org: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c8

This bug can be closed as obsolete/WAI.

We managed to get that we want using VP9E_TEMPORAL_LAYERING_MODE_BYPASS and
setting all the references ourself with VP9E_SVC_SET_REF_CONFIG.
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[ja… via monorail]

Updates:
Status: WontFix

Comment #9 on issue 1580 by ***@google.com: vp9 encoder in SVC flexible
mode and skipped layers can create strange spatial references
https://bugs.chromium.org/p/webm/issues/detail?id=1580#c9

(No comment was entered for this change.)
--
You received this message because:
1. The project was configured to send all issue notifications to this
address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings