Record videos of the test procedure

With this blog series I want to show how important functions can be built into selenium by using simple means. In the first part I introduced what Selenium 4 brings and how to use screenshots. In the second part we will create a video of the test execution. I try to evaluate the approaches according to their added value (The Good) and their challenges (The Bad) and to give useful hints (… and the Useful).

Why?

But at the beginning we will briefly ask ourselves the question: Why record a video or screencast of the test performance?

With a video we have a recording of the entire test run. Unlike with a screenshot, we can not only record the test result, but we can also trace the path to it.

Like screenshots, the videos can also be used for debugging and point out problems during the test run. Therefore, as with the screenshot, it makes sense to create videos only when problems occur. Following this approach, the video functionality should be extended by a flexible and global switch that can be set as needed.

But the videos can also be used to document the test results. In some projects a detailed documentation is even mandatory, because here legal or other requirements have to be met.

Recording video made easy

Since I use Java Selenium, my first approach was to record videos with the “in-house” functions of Java or to use a suitable framework. My first choice was the MonteCC framework, because it allowed me to record the screen during the test execution with the help of only two methods. In one method video recording is started before the test and in one method the recording is stopped after the test and the video is stored in the appropriate directory.

private ScreenRecorder screenRecorder;
 
@BeforeEach
public void beforeTest() {
 
    GraphicsConfiguration gc = GraphicsEnvironment
            .getLocalGraphicsEnvironment()
            .getDefaultScreenDevice()
            .getDefaultConfiguration();
 
    try {
        this.screenRecorder = new ScreenRecorder(gc,
                new Format(MediaTypeKey, MediaType.FILE, MimeTypeKey, MIME_AVI),
                new Format(MediaTypeKey, MediaType.VIDEO, EncodingKey, ENCODING_AVI_TECHSMITH_SCREEN_CAPTURE,
                        CompressorNameKey, ENCODING_AVI_TECHSMITH_SCREEN_CAPTURE,
                        DepthKey, 24, FrameRateKey, Rational.valueOf(15),
                        QualityKey, 1.0f,
                        KeyFrameIntervalKey, 15 * 60),
                new Format(MediaTypeKey, MediaType.VIDEO, EncodingKey,"black",
                        FrameRateKey, Rational.valueOf(30)),
                null);
        this.screenRecorder.start();
 
    } catch (Exception e) {
        System.out.println("screenRecorder.start " + e.getMessage());
    }
}
 
@AfterEach
public void afterTest()  {
 
    try {
        this.screenRecorder.stop();
        List createdMovieFiles = screenRecorder.getCreatedMovieFiles();
        for (Object movie: createdMovieFiles) {
            File m = (File)movie;
            System.out.println("New movie created: " + m.getAbsolutePath());
        }
 
    } catch (IOException ioe) {
        System.out.println("screenRecorder.stop " + ioe.getMessage());
    }
}

To make sure this happens before each test, we use the annotations of JUnit:

@Test

The method with the annotation @Test is a test case and is started during test execution and the result is recorded in the test log.

@BeforeEach

This method is called before each test execution (see @Test).

@AfterEach

This method is called after each test execution (see @Test).

If you need a complete record of all test cases, you can also use the annotations @Before and @After, which are called before all or after all tests of an execution definition (test class).

Major disadvantages of MonteCC are that the last release was more than six years ago and that the videos are in QuickTime format.

JavaCV (https://github.com/bytedeco/javacv) offered itself as an alternative. The library is actively maintained in GitHub and stores videos in MPEG format. Again, there are two methods of video creation that can be called before and after the test. But JavaCV requires additional methods because screenshots are taken at short intervals parallel to the test run and then assembled into a video after the test. A detailed instruction can be found under the following link: https://cooltrickshome.blogspot.com/2016/12/create-your-own-free-screen-recorder.html.

I have made the following changes for use in automated testing:

/**
 *
 * https://github.com/bytedeco/javacv
 * https://cooltrickshome.blogspot.com/2016/12/create-your-own-free-screen-recorder.html
 */
public class RecordSeleniumJavaCV {
 
    // The WebDriver is a tool for writing automated tests of websites.
    FirefoxDriver driver;
 
    public static boolean videoComplete=false;
    public static String inputImageDir= "videos" + File.separator + "inputImgFolder"+File.separator;
    public static String inputImgExt="png";
    public static String outputVideoDir= "videos" + File.separator;
    public static String outputVideo;
    public static int counter=0;
    public static int imgProcessed=0;
    public static FFmpegFrameRecorder recorder=null;
    public static int videoWidth=1920;
    public static int videoHeight=1080;
    public static int videoFrameRate=3;
    public static int videoQuality=0; // 0 is the max quality
    public static int videoBitRate=9000;
    public static String videoFormat="mp4";
    public static int videoCodec=avcodec.AV_CODEC_ID_MPEG4;
    public static Thread t1=null;
    public static Thread t2=null;
    public static boolean isRegionSelected=false;
    public static int c1=0;
    public static int c2=0;
    public static int c3=0;
    public static int c4=0;
 
    /**
     * Explanation:
     * 1) videoComplete variables tells if user has stopped the recording or not.
     * 2) inputImageDir defines the input directory where screenshots will be stored which would be utilized by the video thread
     * 3) inputImgExt denotes the extension of the image taken for screenshot.
     * 4) outputVideo is the name of the recorded video file
     * 5) counter is used for numbering the screenshots when stored in input directory.
     * 6) recorder is used for starting and stopping the video recording
     * 7) videoWidth, videoFrameRate etc define output video param
     * 8) If user wants to record only a selected region then c1,c2,c3,c4 denotes the coordinate
     *
     * @return
     * @throws Exception
     */
    public static FFmpegFrameRecorder getRecorder() throws Exception
    {
        if(recorder!=null)
        {
            return recorder;
        }
        recorder = new FFmpegFrameRecorder(outputVideo,videoWidth,videoHeight);
        try
        {
            recorder.setFrameRate(videoFrameRate);
            recorder.setVideoCodec(videoCodec);
            recorder.setVideoBitrate(videoBitRate);
            recorder.setFormat(videoFormat);
            recorder.setVideoQuality(videoQuality); // maximum quality
            recorder.start();
        }
        catch(Exception e)
        {
            System.out.println("Exception while starting the recorder object "+e.getMessage());
            throw new Exception("Unable to start recorder");
        }
        return recorder;
    }
 
    /**
     * Explanation:
     * 1) This method is used to get the Recorder object.
     * 2) We create an object of FFmpegFrameRecorder named "Recorder" and then set all its video parameters.
     * 3) Lastly we start the recorder and then return the object.
     *
     * @return
     * @throws Exception
     */
    public static Robot getRobot() throws Exception
    {
        Robot r=null;
        try {
            r = new Robot();
            return r;
        } catch (AWTException e) {
            System.out.println("Issue while initiating Robot object "+e.getMessage());
            throw new Exception("Issue while initiating Robot object");
        }
    }
 
    /**
     * Explanation:
     * 1) Two threads are started in this module when user starts the recording
     * 2) First thread calls the takeScreenshot module which keeps on taking screenshot of user screen and saves them on local disk.
     * 3) Second thread calls the prepareVideo which monitors the screenshot created in step 2 and add them continuously on the video.
     *
     * @param r
     */
    public static void takeScreenshot(Robot r)
    {
        Dimension size = Toolkit.getDefaultToolkit().getScreenSize();
        Rectangle rec=new Rectangle(size);
        if(isRegionSelected)
        {
            rec=new Rectangle(c1, c2, c3-c1, c4-c2);
        }
        while(!videoComplete)
        {
            counter++;
            BufferedImage img = r.createScreenCapture(rec);
            try {
                ImageIO.write(img, inputImgExt, new File(inputImageDir+counter+"."+inputImgExt));
            } catch (IOException e) {
                System.out.println("Got an issue while writing the screenshot to disk "+e.getMessage());
                counter--;
            }
        }
    }
 
    /**
     * Explanation:
     * 1) If user has selected a region for recording then we set the rectangle with the coordinate value of c1,c2,c3,c4. Otherwise we set the rectangle to be full screen
     * 2) Now we run a loop until videoComplete is false (remains false until user press stop recording.
     * 3) Now we capture the region and write the same to the input image directory.
     * 4) So when user starts the recording this method keeps on taking screenshot and saves them into disk.
     *
     */
    public static void prepareVideo()
    {
        File scanFolder=new File(inputImageDir);
        while(!videoComplete)
        {
            File[] inputFiles=scanFolder.listFiles();
            try {
                getRobot().delay(500);
            } catch (Exception e) {
            }
            //for(int i=0;i<scanFolder.list().length;i++)
            for(int i=0;i<inputFiles.length;i++)
            {
                //imgProcessed++;
                addImageToVideo(inputFiles[i].getAbsolutePath());
                //String imgToAdd=scanFolder.getAbsolutePath()+File.separator+imgProcessed+"."+inputImgExt;
                //addImageToVideo(imgToAdd);
                //new File(imgToAdd).delete();
                inputFiles[i].delete();
            }
        }
        File[] inputFiles=scanFolder.listFiles();
        for(int i=0;i<inputFiles.length;i++)
        {
            addImageToVideo(inputFiles[i].getAbsolutePath());
            inputFiles[i].delete();
        }
    }
 
    /**
     * Explanation:
     * 1) cvLoadImage is used to load the image passed as argument
     * 2) We call the convert method to convert the image to frame which could be used by the recorder
     * 3) We pass the frame obtained in step 2 and add the same in the recorder by calling the record method.
     *
     * @return
     */
    public static OpenCVFrameConverter.ToIplImage getFrameConverter()
    {
        OpenCVFrameConverter.ToIplImage grabberConverter = new OpenCVFrameConverter.ToIplImage();
        return grabberConverter;
    }
 
    /**
     * Explanation:
     * 1) We start a loop which will run until video complete is set true (done only when user press stop recording)
     * 2) We keep on monitoring the input  Image directory
     * 3) We traverse each file found in the input image directory and add those images to video using the addImageToVideo method. After the image has been added we delete the image
     * 4) Using the loop in step1 we keep on repeating step 2 and 3 so that each image gets added to video. We added a delay of 500ms so that this module does not picks a half created image from the takeScreenshot module
     * 5) When user press stop recording the loop gets broken. Now we finally traverse the input image directory and add the remaining images to video.
     *
     * @param imgPath
     */
    public static void addImageToVideo(String imgPath)
    {
        try {
            getRecorder().record(getFrameConverter().convert(cvLoadImage(imgPath)));
        } catch (Exception e) {
            System.out.println("Exception while adding image to video "+e.getMessage());
        }
    }
 
    /**
     * Explanation:
     * 1) We make a JFrame with the button for staring and stopping the recording. One more button is added for allowing user to record only a selected portion of screen
     * 2) If user clicks to select only certain region then we call a class CropRegion method getImage which helps in retrieving the coordinate of the region selected by user and update the same in variable c1,c2,c3,c4
     * 3) If user clicks on start recording then startRecording method is called
     * 4) If user clicks on stoprecording then stopRecording method is called
     */
    @BeforeEach
    public void beforeTest() {
 
        System.out.println("this.screenRecorder.start()");
 
        SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMdd_HHmmss");
        String timestamp  = dateFormat.format(new Date());
 
        outputVideo= outputVideoDir + "recording_" + timestamp + ".mp4";
 
        try {
            t1=new Thread()
            {
                public void run() {
                    try {
                        takeScreenshot(getRobot());
                    } catch (Exception e) {
                        System.out.println("Cannot make robot object, Exiting program "+e.getMessage());
                        System.exit(0);
                    }
                }
            };
            t2=new Thread()
            {
                public void run() {
                    prepareVideo();
                }
            };
            t1.start();
            t2.start();
            System.out.println("Started recording at "+new Date());
 
 
        } catch (Exception e) {
            System.out.println("screenRecorder.start " + e.getMessage());
        }
    }
 
    @AfterEach
    public void afterTest()  {
 
        System.out.println("this.screenRecorder.stop()");
 
        try {
            videoComplete=true;
            System.out.println("Stopping recording at "+new Date());
            t1.join();
            System.out.println("Screenshot thread complete");
            t2.join();
            System.out.println("Video maker thread complete");
            getRecorder().stop();
            System.out.println("Recording has been saved successfully at "+new File(outputVideo).getAbsolutePath());
 
        } catch (Exception e) {
            System.out.println("screenRecorder.stop " + e.getMessage());
        }
    }
 
    @Test
    public void testMe() {
         
        //toDo
 
    }
 
}

Is there a catch?

So, we now have several possibilities to record the automatic test run in a video in order to be able to reproduce it in case of problems or to fulfil the documentation obligations.

But recording during the test run is associated with a risk that the tester / test automator should keep in mind: It is an interference with the normal test run, because we might change the timing. Our modified test run with video recording may behave differently than a test run without video recording.

Videos of the test procedure