Last updated January 22, 2019 at 3:30PM.

NanoDB uses the TestNG framework to automate the testing of NanoDB. Following the pattern specified by Maven, the source-code for all tests live in the src/test/java subdirectory of NanoDB, and are in edu.caltech.test.nanodb.* packages. Tests are put in packages similar to the packages of the code that they exercise, although this is not always exactly how things are done.

In general, these are the goals for unit tests:

Obviously, sometimes the functionality being exercised by a unit test includes a lot of code, because the functionality itself is complicated. An example would be adding a record to a B+ tree tuple file, which requires many steps. However, as long as there are also unit tests that exercise all of the functionality that the B+ tree is built upon (e.g. Schema, DBFile, TupleComparator, etc.), then those tests will also help constrain where a specific failure may be occurring.

A test's scope can further be constrained by the input data being used, etc., etc. Ideally, each test will exercise a limited amount of functionality, so that a failure will be relatively easy to track down.

Configuring Which Tests to Run

TestNG allows tests to be grouped together, and then specific groups of tests can be included or excluded when a build is performed. So that you know what to look for, the Java annotation used by TestNG is as follows:

@Test(groups={"framework"})
public class TestSchema {
    ... // Code to exercise the Schema class
}

The groups to run are specified in the Maven pom.xml file, in the section containing Surefire configuration. Look for this in the NanoDB pom.xml file:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    ...

    <configuration>
        ...

        <groups>framework,parser</groups>
    </configuration>
</plugin>

You can put additional testing groups into the <groups> section. For example, to include the HW1 and HW2 tests, you would update the above configuration to:

        <groups>framework,parser,hw1,hw2</groups>

Viewing Automated Test Results

Of course, when you run mvn test or mvn project, you will see the result of running the automated test suite in the console output, for example:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
[INFO] Tests run: 137, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.135 s - in TestSuite

If any tests fail then you have a couple of options. If you are running your build from IntelliJ IDEA, you can click on the failed tests in the build output, and IDEA will take you to the source of the test that failed. Alternately, if you are running your build from the command-line, or you just want to see all of your test results in one place, you can navigate to the generated Surefire reports to see what passed and what failed. The reports are stored at this path in your project: target/surefire-reports/index.html. Of course, you can open this file in a web browser and navigate through the test report.

Viewing Automated Test Coverage

It can be helpful to see how much code is actually exercised by a given set of automated tests. Actually, this can be a huge red-herring, and test coverage statistics must be considered with a healthy dose of salt. What is more useful is seeing what code is not exercised by a set of tests, so that you can write additional tests to exercise that code.

NanoDB includes the JaCoCo library to examine test coverage. ("JaCoCo" stands for "Java Code Coverage.") JaCoCo works by instrumenting the NanoDB codebase and recording what parts of NanoDB are executed by the tests that are run. This information can then be used to generate code-coverage reports.

The NanoDB Maven configuration uses JaCoCo to instrument the TestNG suite, and then generates a code-coverage report after the tests have completed. You can see this code-coverage report at this path in your project: target/coverage-reports/index.html. As with the test results, you can open this file in a web browser and navigate through the code-coverage report.

It is very easy to write tests that have a very large code coverage: Just have the test do some complicated thing that involves a lot of the codebase, but has a simple answer, and there you go. For example, you could write a test that starts the NanoDB server, verifies that "SELECT 3 + 4 AS answer;" gives 7, and you would cover a lot of code. However, this defeats one of the main goals of unit-tests, which is to exercise the smallest possible units of functionality in the codebase so that failures are easy to detect and repair.

Test code-coverage is a valuable aid in achieving the above, but just increasing coverage numbers for the sake of increasing them is not a useful approach.

Running a Test in the IntelliJ IDEA Debugger

If you need to run tests in the IntelliJ IDEA debugger, it is mostly straightforward to get this working. The only issue is that any breakpoints you set in your code will be ignored under the default Maven configuration; you must make a minor tweak to the configuration to get breakpoints working. Here are some simple steps for getting this working.

  1. First, run the Maven "test" target from inside IntelliJ IDEA. You can do this by expanding the "Maven" tab on the right side of the UI, then double-clicking the "test" item under the "Lifecycle" group.

    Doing this will automatically create a "nanodb-server [test]" configuration that will be used when you run or debug tests within IntelliJ IDEA.

    (If you have already run the Maven "test" target from inside IntelliJ IDEA then you can skip over this step.)

  2. Choose the "Run -> Edit Configurations..." menu items to edit the configuration that should exist after running the previous step. A window should pop up, and in the top left you should see a "Maven" group with an entry "nanodb-server [test]". Select that entry; its configuration will appear in the right side of the window.

    In the right side of the window, select the "Runner" tab. The "Use project settings" checkbox will be checked; un-check it, then add this to the "VM Options" text box: -DforkMode=never

    (Note: You will likely get a deprecation warning about the forkMode property, but the newer version -DforkCount=0 doesn't seem to work.)

    (Note: Disabling fork-mode will also disable code-coverage data collection and reporting.)

  3. Now you should be able to set breakpoints in your server code and/or test code, then right-click the Maven "test" target and select the menu item "Debug 'nanodb-server [test]'". The debugger should stop when it hits your breakpoints.