Hermetic Testing?

Q. What are hermetic tests?

A. Tests should be hermetic: that is, they ought to access only those resources on which they have a declared dependency. If tests are not properly hermetic then they do not give historically reproducible results. This could be a significant problem for culprit finding (determining which change broke a test), release engineering auditability, and resource isolation of tests (automated testing frameworks ought not DDOS a server because some tests happen to talk to it).

Q. How Hermetic servers are used for E2E testing?
A. Google uses this trick to  design their end-to-end tests.
What is a Hermetic Server? The short definition would be a “server in a box”. If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! \

Q. How do you design a Hermetic server?
A. 1. All connections to other servers are injected into the server at runtime using a suitable form of dependency injection such as commandline flags or Guice.
2. All required static files are bundled in the server binary.
3. If the server talks to a datastore, make sure the datastore can be faked with data files or in-memory implementations.
4. Make sure those connection points which our test won’t exercise have appropriate fakes or mocks to verify this non-interaction.
5. Provide modules to easily populate datastores with test data.
6. Provide logging modules that can help trace the request/response path as it passes through the SUT.

An Interesting slide deck by Spotify on "Hermetic environment for your functional tests"

Hermetic environment for your functional tests from Andrii Dzynia

Sources:

http://bazel.io/docs/test-encyclopedia.html

http://googletesting.blogspot.com.au/2012/10/hermetic-servers.html