Developer Manual

Testing

Requirements before merging changes in git include:

  • • Tests must pass. (Merge blocking can be enforced eg. on github.)

  • • The code must be in the standard style, which will be at least partly enforced as part of the automated testing.

  • • Documentation must be updated or added to reflect changes in the code.

Source code testing

Testing of code is essential to ensure correctness, reduce incidents of accidental breakage or regression of code features, and enable code changes to be made with confidence. These tests must be automated, and as far as possible be “unit" tests, which test isolated components of the code. A strictly Test Driven Development (TDD) approach is not always appropriate, but encouraging incremental development and testing in small pieces has several advantages in terms of the resulting code structure and maintainability:

  • 1. It encourages the writing of code which has “clean" interfaces ie. a well defined set of inputs and outputs, with minimal side channels (eg. global state).

  • 2. Having to test components individually discourages strong coupling between code, because then these dependency components have to be “mocked" up in testing.

  • 3. Good code test coverage makes later maintenance, modification and refactoring of the code easier. The tests also function as a type of documentation of the intended use of the code, and also of the corner-cases which may not be obvious to a new user or developer.

The most important types of tests are for correctness. These can use standard services such as github actions, Travis etc. Performance is however a crucial property of the code, and should also be monitored.

Performance testing

It is useful to include timing information in test output, which is then contained in the test logs. This is valuable as a quick way for developers to observe the impact of changes on performance. It is however not very accurate, especially under virtual machines on shared hardware as is typical for testing services. These tests also only typically use a small number of processors (less than four), usually without accelerator support, making them of limited use in evaluating performance of high performance code for the Exascale.

Periodic testing of code versions on a range of hardware will be needed to monitor performance, and catch performance regressions. This could be carried out by a researcher, but the possibility of automating this process and making use of services such as Amazon AWS HPC and GPU servers. Studies carried to date indicate a lack of appropriate software for ensuring performance portability and a consequent need at least to enhance existing packages.

Object Identification

Douglass ref. [106, § 5] has a description of object analysis which is well-suited project NEPTUNE. His approach is to take the use cases, which for this purpose should include the proxyapps separately, and treat them carefully one after the other using the strategies indicated in Table 9.1. Each proxyapp should be carefully analysed and classes produced from the list of objects before proceeding to the next.

Table 9.1: Key Strategies for Object Identification. After Table 5.1 from ref. [106], slightly amended. All the strategies except the last, are concerned with identifying the objects listed.

Strategy

Description

Nouns

Used to gain a first-cut object list, the analyst underlines each noun or noun phrase in the problem statement and evaluates it as a potential object, class, or attribute.

Causal agents

Identify the sources of actions, events, and messages; includes the coordinators of actions.

Services (passive contributors)

Identify the targets of actions, events, and messages as well as entities that passively provide services when requested.

Messages and information flow

Messages must have an object that sends them and an object that receives them as well as, possibly other objects that process the information contained in the messages. There are many ways to identify the objects within a collaboration.

Real-world items

Real-world items are entities that exist in the real world, but are not necessarily electronic devices. Examples include objects such as gases, forces, blanket modules, etc.

Physical devices

Physical devices include the sensors and actuators provided by the system as well as the electronic devices they monitor or control. The resulting objects are almost always the interfaces to the devices. Note: this is a special kind of “Identify real-world items".

Key concepts

Key concepts may be modeled as objects. Physical theories exist only conceptually, but are critical scientific objects. Frequency bins for an on-line autocorrelator may also be objects. Contrast with the “identify real-world items" strategy.

Transactions

Transactions are finite instances of interactions between objects that persist for some significant period of time. An example is queued data.

Persistent information

Information that must persist for significant periods of time may be objects or attributes. This persistence may extend beyond the power cycling of the device.

Visual elements

User-interface elements that display data are objects within the user-interface domain such as windows, buttons, scroll bars, menus, histograms, waveforms, icons, bitmaps, and fonts.

Control elements

Control elements are objects that provide the interface for the user (or some external device) to control system behavior.

Apply scenarios

Walk through scenarios using the identified objects. Missing objects will become apparent when required actions cannot be achieved with existing objects and relations.