When a flip-flop samples an input that is changing during its aperture, the output Q may momentarily take on a voltage between 0 and VDD that is in the forbidden zone. This is called a metastable state.
It can be thought of as a ball on top of a curve (metastable) which will eventually settle to either sides (stable). Every bistable device has a metastable state between the two stable states.
Aperture time is defined as the sum of the setup time and hold time, this is the total time for which the input must remain stable.
Let’s consider the following example with 3 cases:
Case I: Input ‘D’ changes before the aperture time, stable zone
Case II: Input ‘D’ changes after the aperture time, stable zone
Case III: Input ‘D’ changes during the aperture time, metastable zone. In this case the input violates the setup time & the hold time constraint and output is undefined.
Eventually, the flip-flop will resolve the output to a stable state of either 0 or 1. However, the resolution time required to reach the stable state is unbounded.
A designer should avoid any metastable states in the system and be mindful of not violating the aperture time for flip-flops used in the system.
Source: Digital Design & Computer Architecture by Harris & Harris