Generating a variable frame rate video from image files

EDIT: Please note this question has nothing to do with VBR encoding. I want to generate a video from images at 120 frames per second while speeding up or slowing down the input stream by repeating/omitting frames.

I have a list of image files from which a video needs to be generated using a playback control script.

var files = new DirectoryInfo(@"C:\Images")
    .GetFiles(@"*.png", SearchOption.TopDirectoryOnly)
    .OrderBy(f => f.FullName)
    .ToList();

The files are numbered from 1 to n as follows:

00000001.png
00000002.png
.......
99999998.png
99999999.png

There is a user-created playback control script as follows:

Sample Script
    Step 1
        Direction: Forward
        Duration: 10 Seconds
        Speed: 60 FPS (50%)
    Step 3
        Direction: Forward
        Duration: 10 Seconds
        Speed: 120 FPS (100%)
    Step 2
        Direction: Backward
        Duration: 10 Seconds
        Speed: 30 FPS (25%)
    Step n
        ....................

The images were captured at 120 frames per second so the resulting video should always be generated at 120 FPS. The video will achieve a variable FPS effect by skipping or repeating frames based on the script.

The above script generates a 30 second video and I'd like to create a new List<FileInfo> that contains (120 x 30 = 3,600) entries including repetitions and omissions.

The original file list is sorted by the order of frame arrival. For some reason though, I can't get a grip on how to write the algorithm. I've tried keeping an index to the current frame being processed but have not been able to control it using repetition/omission counts or time stamps.

Just looking for a nudge in the right direction.