Hermetic Testing?


Q. What are hermetic tests?

A. Tests should be hermetic: that is, they ought to access only those resources on which they have a declared dependency. If tests are not properly hermetic then they do not give historically reproducible results. This could be a significant problem for culprit finding (determining which change broke a test), release engineering auditability, and resource isolation of tests (automated testing frameworks ought not DDOS a server because some tests happen to talk to it).

Q. How Hermetic servers are used for E2E testing?
A. Google uses this trick to  design their end-to-end tests.
What is a Hermetic Server? The short definition would be a “server in a box”. If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! \

Q. How do you design a Hermetic server?
A. 1. All connections to other servers are injected into the server at runtime using a suitable form of dependency injection such as commandline flags or Guice.
2. All required static files are bundled in the server binary.
3. If the server talks to a datastore, make sure the datastore can be faked with data files or in-memory implementations.
4. Make sure those connection points which our test won’t exercise have appropriate fakes or mocks to verify this non-interaction.
5. Provide modules to easily populate datastores with test data.
6. Provide logging modules that can help trace the request/response path as it passes through the SUT.

An Interesting slide deck by Spotify on "Hermetic environment for your functional tests"


Sources:
http://bazel.io/docs/test-encyclopedia.html
http://googletesting.blogspot.com.au/2012/10/hermetic-servers.html

Comments

Popular posts from this blog

XPATH for IE / internet explorer

Website's underlying technology for automation

All time good software testing books!