Jetty Project and TCK

The Jetty project has a long history of participating in the standardization of EExx (previously JEE) specifications such as Servlet and Websocket.

Jakarta renaming

After the donation of TCK source code by Oracle to Eclipse Foundation, the EE group has decided to change the historical Java package names from javax.servlet or javax.websocket (etc..) to jakarta.servlet or jakarta.websocket (etc..). As we wanted to be able to validate our EE9 implementation, we have participated in the big change by contributing two Pull Requests doing the migration (Servlet/Websocket TCK https://github.com/jakartaee/platform-tck/pull/257) and (JSP, JSF, JAX* https://github.com/jakartaee/platform-tck/pull/276). Those Pull Requests have been a lot of work and testing as 1746 and 857 files of old/legacy code have been modified.

Refactoring of Servlet TCK

The original TCK (still the current main branch as of the date of writing) is based on a technology called JavaTest Harness (more details here). The problem with this technology is the slowness of test execution and the difficulties to debug what is happening.

After some discussion,  a decision was made to rewrite All TCKs using Junit5 (or TestNG) and use Arquillian.

Starting with those requirements, it has been a long journey to refactor the Servlet TCK.

Originally, the Servlet TCK is a cascade of Ant builds which are packaging 210 war files to be deployed in the target Servlet container to be validated using the JavaTest Harness framework. This means a jvm is restarted 210 times to run all the tests and the TCK has 1691 tests.

On the Jetty CI, the daily build testing against the legacy TCK takes around 1h10min (Jenkins Job) and it’s not easy to set up such a run locally.

After the refactoring to use Arquillian and embedded Jetty instance, the similar Jenkins Job takes around 17 minutes (including a full rebuild of Arquillian Container Jetty, TCK servlet and Jetty 12).

Maven Build

As we were starting a big refactor with Jetty 12, having those 1691 tests was convenient to test all the changes but definitely too slow to run locally. So thus began the journey of rewriting the Servlet TCK.

The first job was to have a Maven build for the whole TCK, the branch called tckrefactor has been started to refactor the single source directory to multiple Maven modules with separation of concern. But as you can imagine this took some time to move Java files to the correct Maven modules without breaking some (circular?) dependencies that a single source directory was allowing.

Junit

Then we had to find the pattern (e.g. what do we have to do exactly?). As explained the TCK is/was a collection of wars and classes running using JavaTest Harness.

The tests were using the Javadoc taglet to discover the tests to be executed. The most complicated example is a mix using the Javadoc taglet such as:

public class Client extends secformClient {
......
  /*
   * @testName: test1
   *
   * @assertion_ids: Servlet:SPEC:142; JavaEE:SPEC:21
   *
   * @test_Strategy: .....
   *
   */

  /*
   * @testName: test1_anno
   *
   * @assertion_ids: Servlet:SPEC:142; JavaEE:SPEC:21; Servlet:SPEC:290;
   * Servlet:SPEC:291; Servlet:SPEC:293; Servlet:SPEC:296; Servlet:SPEC:297;
   * Servlet:SPEC:298;
   *
   * @test_Strategy: ......
   *
   */
  public void test1_anno() throws Fault {
    // save off pageSec so that we can reuse it
    String tempPageSec = pageSec;
    pageSec = "/servlet_sec_secform_web/ServletSecAnnoTest";

    try {
      super.test1();
    } catch (Fault e) {
      throw e;
    } finally {
      // reset pageSec to orig value
      pageSec = tempPageSec;
    }
  }

This test class when executed by the Java TestHarness framework will run super.test1 and test1_anno methods. With moving to junit5 we need to add the annotation @Test to all methods which need to run.

This means in this case we need to execute the test1 method in the superclass and the method test1_anno in the current class. The solution could be to mark test1_anno and the method test1 in the parent class with the annotation @Test but we cannot touch the parent class otherwise every class inheriting from this will this test method which is not expected and was not the behavior of the TestHarness framework.

The solution is to mark it with @Test annotation the method test1_anno and modify the current source code to add the simple annotated method such as:

@Test
public void test1() {
    super.test1();
}

With more than 800 classes to touch this was not possible so we developed an Apache Maven plugin to achieve this: https://github.com/jetty-project/tck-extract-tests-maven-plugin

Arquillian

As mentioned above, the original TCK was based on Apache Ant building 212 war files to deploy. This means 212 Java files have to be modified to include the generation of the Arquillian WebArchive. Sadly nothing magical here as every build.xml was different and no way to find a pattern to automate this.

The solution was: Roll Up your sleeves™

Given a build.xml such as:

  <property name="app.name"  value="servlet_spec_errorpage" />
  <property name="content.file" value="HTMLErrorPage.html"/>
  <property name="web.war.classes"
            value="com/sun/ts/tests/servlet/common/servlets/HttpTCKServlet.class,
                   com/sun/ts/tests/servlet/common/util/Data.class"/>

  <target name="compile">
    <ts.javac includes="${pkg.dir}/**/*.java,
                        com/sun/ts/tests/common/webclient/validation/**/*.java,
                        com/sun/ts/tests/servlet/common/servlets/**/*.java,
                        com/sun/ts/tests/servlet/common/util/**/*.java"/>
  </target>

was rewritten using Arquillian WebArchive generation in Java code (some manual checks had to be done as some **/**/** were too “generous”) to:

public static WebArchive getTestArchive() throws Exception {
    return ShrinkWrap.create(WebArchive.class, "servlet_spec_errorpage_web.war")
            .addAsWebResource("spec/errorpage/HTMLErrorPage.html","HTMLErrorPage.html")
            .addAsLibraries(CommonServlets.getCommonServletsArchive())
            .addClasses(SecondServletErrorPage.class, ServletErrorPage.class, TestException.class, TestServlet.class,
                    WrappedException.class)
            .setWebXML(URLClient.class.getResource("servlet_spec_errorpage_web.xml"));
  }

This work has taken a lot of time and resources, and generated a quite big Pull Request: https://github.com/jakartaee/platform-tck/pull/912

With all of these changes, as long as your container has Arquillian support it’s very easy to run the Servlet TCK using Apache Maven Surefire with a configuration to pick up all the tests (a sample for Jetty is https://github.com/jetty-project/servlet-tck-run/blob/jetty-12-ee10/pom.xml).

Last but not least as the Jetty Arquilian container is embeded, it is now possible to debug everything (TCK code and Jetty code) using an IDE.

You can run even more TCK by adding them to the Apache Maven Surefire configuration:

        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>${surefire.version}</version>
          <configuration>
            ....
            <includes>
              <include>**/URLClient*</include>
              <include>**/Client*</include>
            </includes>
            <excludes>
              <exclude>**/BaseUrlClient</exclude>
            </excludes>
            <dependenciesToScan>
              <dependenciesToScan>jakartatck:servlet</dependenciesToScan>
            </dependenciesToScan>
          </configuration>
        </plugin>

The next steps are now to have other TCKs converted to Junit/Arquillian. Both Websocket, JSP and EL are works in progress.