Introduction

It is a python script for analyzing performance of a C++ program, including time and memory cost. It is useful for Data Structure and Algorithm course.

You need to provide:

data.cpp - the program to generate data for testing
std.cpp - the standard program that outputs the right answer
main.cpp - the program to test

Implementation

It is gnu-time that provides the main functions, i.e. measuring the time and memory cost.

Follow the document carefully to install it. The common issue is that you should use the real gnu-time, rather than "time" provided by the shell, which is mentioned in the doc. By the way, use --help option to learn how to use it.

For macOS users like me, follow this for installation: https://formulae.brew.sh/formula/gnu-time .

Note that the -f (or--format) option is useful to get outputs we like.

Having gnu-time, we can write the python script below to test the program with increasing data. Use os.popen to get the outputs of gnu-time, and then print the results in whatever formats. For example, a table-like format is convenient for further processing the results with Excel, like plotting.

Read the codes to know how to use it. :)

import sys
import os

def check_ans(output_file1, output_file2):
    with open(output_file1) as f:
        output1 = f.read()
    with open(output_file2) as f:
        output2 = f.read()
    return output1 == output2

if __name__ == '__main__':
    os.system('g++ data.cpp -O3 -std=c++14 -o data')
    os.system('g++ std.cpp -O2 -std=c++14 -o std')
    os.system('g++ main.cpp -O2 -std=c++14 -o main')

    input_data  =   'input.txt'
    ans_txt     =   'ans.txt'
    output_txt  =   'output.txt'

    scale = 10000
    step = 10000
    loop = 10
    if len(sys.argv) >= 2:
        scale = int(sys.argv[1])
        if len(sys.argv) >= 3:
            step = int(sys.argv[2])
            if len(sys.argv) >= 4:
                loop = int(sys.argv[3])

    print('\tscale\t\tProg Time(s)\tProg Mem(KB)\tStd Time(s)\tStd Mem(KB)')
    for i in range(0, loop):

        os.system('./data {} > {}'.format(scale, input_data))
        print('\t' + str(scale) + '\t\t', end='')

        prog_info = os.popen(
            '(gtime -f"%e\t\t%M\t\t" ./main < {} > {}) 2>&1'.format(input_data, output_txt)
        ).readlines()
        print(prog_info[0][:-1], end='')

        std_info = os.popen(
            '(gtime -f"%e\t\t%M\t\t" ./std < {} > {}) 2>&1'.format(input_data, ans_txt)
        ).readlines()
        print(std_info[0][:-1], end='\n')

        not_same = os.system('diff -q {} {}'.format(ans_txt, output_txt))
        if not_same:
            print('WA!')
            break
        else:
            scale += step

Then we can get things like:

$ python tester.py 100000 100000 30

    scale        Prog Time(s)    Prog Mem(KB)    Std Time(s)    Std Mem(KB)
    100000        0.04        1952        0.04        2412
    200000        0.09        2664        0.09        4144
    300000        0.14        3616        0.15        5040
    400000        0.22        5176        0.21        7304
    500000        0.27        5724        0.26        8260
    600000        0.29        7152        0.32        9976
    700000        0.35        9408        0.38        12104
    800000        0.42        10636        0.44        13028
    900000        0.46        9704        0.52        14436
    1000000        0.54        12020        0.60        17308