Rust Design-for-Testability: a survey

Rust logo What can we do when designing Rust code to make it easier to test? This is a survey of everything I could find1 about testing Rust with a particular focus on design for testability for correctness. Some of the articles show multiple things to do on a worked example, some are more focused on a particular trick.

There doesn’t seem to be a single place that describes all the testing ideas: it is scattered across book chapters, blog articles, medium articles, etc. but here are the main sources that I have found.

I am not going to try to summarize what these sources say: the rest of this post is a list of some common / interesting topics and which of these sources describe it in more detail.

Although my main focus is on how to write Rust programs so that they are easy to test, I touch on the other half of the problem: how to do the testing.

[I am new to Rust and my own testing habits are somewhat ad-hoc so this is definitely not a recommendation of how to write software by me. I hope it is useful and that you will tell me what I have missed on twitter or by email so that I can update this post. I would love to hear about any team that has published recommendations for design-for-testability.]

Design techniques for improving testability

The sources listed above have a bunch of common suggestions that I explore in more detail below. Many of the sources I found have great discussions so I will not try to repeat their explanations in this document but will link to some of the better discussions of each idea that I found.

Use intermediate data structures

See: Testing sync, Rust book, Testing CLI applications, use the Rust compiler for integration testing

  • Use intermediate data structures to separate deciding what to do from performing the action: allowing tests to check that the right decision is being made and avoiding the need to mock/fake the filesystem, etc.
  • Aggressively use newtypes, structs, enums
  • Parse and validate inputs early (eg convert strings to enums)
  • Also, #[must_use], parse/validate early
  • Writing I/O-Free (Sans-I/O) Protocol Implementations (not Rust specific)

Abstract testable code into separate functions

See: Rust book, Testing CLI applications

  • Cleanly separate command line parsing from code that implements functionality

Abstract I/O and state side effects out of functions

See: Testing CLI applications, Writing fuzzable code, Testing sync

  • Use of std::io::Write trait and writeln! instead of println! (and handle resulting potential error)

Avoid / reduce non-determinism

See: Testing sync, Writing fuzzable code

  • Use Futures with a custom executor to eliminate the non-determinism of threads
  • All randomized testing systems should be fully deterministic and easily reproducible
  • Beware of additional randomness in libraries: e.g., Rust’s HashMap uses randomized hashing to protect against denial of service attacks. This makes testing harder.
  • Determinism enables minimization of random tests (cf. proptest)

Defining correct behavior

See: Writing fuzzable code

Dependency injection and mock testing

See: Rust 2020: Testing, Testable Component Design in Rust

  • Abstraction can be based on Higher order functions or objects
  • Traits and mockall
  • Module mocks using mocktopus
  • shaku is a compile-time dependency injection library that works well with mockall

API design

See: Testing CLI applications, Rust API guidelines, If you use Unsafe, …, Testable Component Design, Writing Correct, Readable and Performant (CRaP)

API design strongly affects testability of that API

  • The component should provide a stable contract composed of traits, structs, and enums
  • Always implement Clone and fmt::Debug for public types
    • types like failure::Error should be converted to something that is cloneable

  • Use ‘newtype’ to let the type system statically test for errors and use one of these to test that this is done correctly

    • compiletest.rs for testing Rust compilations
    • trybuild for testing error messages (eg from proc-macros)
    • lang_tester for testing compilations including, but not limited to, Rust
    • the fuzzy text matcher fm
  • Use #![warn(missing_doc_code_examples)] (and other Rust API guidelines)

Writing tests

See: How to organize your Rust tests, Testing CLI applications, Rust by example,

Having structured your software to enable tests, there are a lot of different tools and libraries to support writing tests.

Test / Behavior driven design (TDD and BDD)

See: Rust testing tricks, From @test to #[test]: Java to Rust

Obviously, there are many, many articles about TDD, BDD, Agile, etc. in the context of Java and other OO languages. The following links are Rust specific but they are a bit random and need to be improved.

  • Laboratory.rs – BDD-inspired test library (todo: are other BDD libraries maybe more popular?)
  • Speculate RSpec inspired testing library
  • Fluent assertions: spectral (last updated 2017)
  • TDD with Rust (2017) – a small example
  • Regression testing (testing against a golden reference) using one of

Specific topics

Note: links in this section are more likely to be out of date.

Code coverage

Testing embedded systems

Concurrency, futures and async

Testing frameworks

  • Serializing Rust tests (github) – annotations to prevent some tests being run in parallel
  • Skeptic – run doctest-like tests on README.md
  • Trust automated test runner: reruns tests when files change
  • Test-case procedural macro to generate tests from test-case annotations
  • Artifact (aka RST) – requirement tracking software where comments in code are linked (in lightweight way) to requirements, specs and tests in a markdown document
  • Fake.rs – interesting #derive option to describe how to generate fake values for structs. Can this be adapted to specify invariants for legal values of a type?

Testing GUIs

Testing APIs

Mutation testing

  • Mutagen – mutation testing tool implemented using procedural macros

Test generation

Mocking libraries

There are a lot of mocking / faking libraries – this is a limited list of what I found.

Error handling

See: CLI applications in Rust, Structuring and using errors in 2020

Not quite about testing – but semi-relevant.

Misc

More information

My interest in Rust testing is a hope/belief that a good way of making formal verification more accessible to software developers is to build on what they are already familiar with: testing. I described this last month.

If you found this article interesting, you might also enjoy these related posts

Discussion of this post

Thanks

Thanks to the following people for adding to the above

  1. By “everything I could find”, I mean that I searched for terms relating to “design for test”, “testability”, “Rust”, etc. and followed links in other blogs. I also searched the archives of “This week in Rust” for the word “test” and followed any promising links. ↩

Written on October 30, 2020.
The opinions expressed are my own views and not my employer's.