Loop the video with SetFFmpegVideoFilters is failing

[Dush27]

Hi Team,

I am trying to loop the video by using below code, but for some reason the looping option seems to fail.

ffmpegMediaSource2_.SetFFmpegVideoFilters(L"loop=loop=-1,crop=322:181:0:149,scale=640:360");

The same looping issue is not observed if I don't use above line of code to crop and scale the video.

Can you please provide your suggestions ?

NOTE: My intention is to crop, scale and perform loop on the video.

Thanks,
Dushyanth

[lukasf]

Hi @Dush27,

for the loop filter, you need to specify all three options:

This would loop the first frame infinitely:
loop=loop=-1:size=1:start=0

If you want to loop the whole video, you need to know the number of frames contained. I can use this with or without scaling. What troubles me is that I see skyrocketing memory usage. Not sure yet what is going on.

Other approach: How about using MediaPlayer.IsLoopingEnabled instead of a filter? The loop filter is generally memory intensive. Looping on MediaPlayer would be the preferred approach. Also, MediaPlayer does not like it when the file plays longer than the duration we set.

[Dush27]

Hi Lukas,

-> I have tried this SetPosition already which works to loop back to start of the file when I don't use SetFFmpegVideoFilters().
Point to be noted is that there is lag(approx 2-3seconds delay for a 20seconds playback) everytime when ReadSample is
performed for first sample when end of stream is reached.Can we avoid this lag by some means? I guess it is the
processing of the file again by ffmpeg?

FFmpegInteropX log without SetFFmpegVideoFilters():

GetNextSample
 - QueuePacket
 - PopPacket
GetNextSample
GetNextPacket reaching EOF
End of stream reached. Enter draining mode.
End of stream reached. No more samples in decoder.
End of stream reached.
DisableStream
Flush
EnableStream
Flush
GetNextSample
 - QueuePacket
 - PopPacket
GetNextSample_

-> If I use SetPosition with SetFFmpegVideoFilters(), loop back to start of the file fails and video is stuck at the last frame.

FFmpegInteropX log with SetFFmpegVideoFilters():

_GetNextSample
 - QueuePacket
 - PopPacket
GetNextSample
GetNextPacket reaching EOF
End of stream reached. Enter draining mode.
End of stream reached. No more samples in decoder.
End of stream reached.
DisableStream
Flush
EnableStream
Flush
GetNextSample
End of stream reached. No more samples in decoder.
End of stream reached.
DisableStream
Flush
EnableStream
Flush
GetNextSample
End of stream reached. No more samples in decoder.
End of stream reached.
DisableStream
Flush
EnableStream_

NOTE: I have used following code to avoid the issue with Filters but i am not able to avoid the delay for the first ReadSample() invocation when the end of stream is reached.

if (flags & MF_SOURCE_READERF_ENDOFSTREAM || sample == nullptr) {
            //Below change is a workaround to loop the video as FFmpegInteropX loop fails 
            //when we apply crop and scale using SetFFmpegVideoFilters
            **ffmpegMediaSource2_.DisableVideoEffects();**
            if (SourceCanSeek()) {
                 winrt::check_hresult(SetPosition(LONGLONG(0)));
                 **ffmpegMediaSource2_.SetFFmpegVideoFilters(wss_.str().c_str());**
            } 
}

Thanks,
Dushyanth

[lukasf]

Thanks for reporting the issue with filters on EOF. I have a fix, please try branch filters-eof.

I don't know where the lag comes from. If I seek to beginning with normal MediaPlayer and start playback, it happens nearly instantly. So maybe it has to do with IFMSourceReader.

[lukasf]

One alternative would be using MediaPlayer in frame server mode:

https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/play-audio-and-video-with-mediaplayer#use-mediaplayer-in-frame-server-mode

You have full HW acceleration and get an event when a new frame is to be rendered. You copy the frame to a surface and forward that to your encoder or whatever you have behind the decoder part. You don't need to care about timestamps because MediaPlayer raises the events in the right moment. Then you can set IsLoopingEnabled=true on the MediaPlayer and it will do smooth and fast looping of the video file. The file you sent plays without any noticeable delay with IsLoopingEnabled.

See DirectXInteropHelper and D3D11VideoSampleProvider on how to interop from IDirect3DSurface <-> IDXGISurface <-> ID3D11Texture2D. You do not need to use Win2D as in the docs. You can use any ID3D11Texture2D as target, I think only ArraySize needs to be 1, otherwise you cannot create a IDirect3DSurface from it.

[Dush27]

Hi Lukas,

Sorry for the late response.

I gave a try with "filters-eof" and below is my observation.
-> I don't see the video playback fail while reaching the end of stream, but the filters which are applied while creating the reader is not getting applied automatically after reaching the end of stream.

I had to apply the filters again while performing SetPosition as below, to apply the filters while reaching the end of the stream as below.

if (flags & MF_SOURCE_READERF_ENDOFSTREAM || sample == nullptr) {
            if (SourceCanSeek()) {
                winrt::check_hresult(SetPosition(LONGLONG(0)));
                ffmpegMediaSource2_.SetFFmpegVideoFilters(wss_.str().c_str());
             }
 }

Regarding your suggestion of MediaPlayer is frame server mode, can we follow this method to integrate with win32 application ?

Thanks,
Dushyanth

[lukasf]

Hi @Dush27, yes frame server mode definitely works in Win32 Desktop apps.

You are right, the fix in filters-eof did not really work. It kept playing, but it silently dropped the filter effect. I read a bit about filters, and it seems that we should not even try to drain them, because they won't be usable after that. I have updated the branch with a new fix, and it looks like this one really works and keeps the filter applied after eof. This one also works with MediaPlayer.IsLoopingEnabled=true.

[Dush27]

Hi @lukasf,

Now the filter branch seems to work fine.

Thanks,
Dushyanth

[lukasf]

Good to hear that.