# A Unified Theory Of Time Frequency Reassignment Form

## Reassigned Spectrogram

The reassigned spectrogram is a special graphic time / frequency representation; especially suited for "fast" spectrograms. Compared with the classic spectrogram (aka 'waterfall') display, reassigned spectrograms can offer better resolution in the time- as well as in the frequency domain.

For example, the following two spectrograms show the same signal, using the same display parameters (same FFT window, same FFT size, same scroll interval...):

Image 1 : Reassigned Spectrogram of a synthetic signal

Image 2 : Classic spectrogram, showing the same signal (a few seconds later)

The reassigned spectrogram shows a much 'sharper' image than the classic spectrogram, as long as the criteria for separability are fulfilled, and the signals have a sufficient signal-to-noise ratio. The principle of time/frequency reassignment is described further below; several in-depth articles about the topic can be found through a websearch.

The display settings, and the configuration of the test signal generator are contained in the configuration 'reassigned_spectrogram_test.usr' . How to load such files into SpecLab is explained here.

The parameters for the two spectrograms shown above were:

- 44100 audio samples per second
- 512-point FFT with HANN window -> 12 millisecond FFT window time, 86 Hz FFT bin width
- 0.73 ms scroll interval (aka 'frame advance') -> 93.75 % overlap between two consecutive FFTs (STFTs)

The test signal was produced with SL's test signal generator. It consisted of..

- A sine wave, frequency modulated with another sine wave.

Modulation frequency: 10 Hz, frequency deviation 1000 Hz . - A 5-Hz sawtooth function

(the sharp falling edge is visible as narrow vertial line in the reassigned spectrogram) - An unmodulated frequency sine wave

(move this a bit higher in frequency so it overlaps with the FM'ed sine, to see what happens without sufficient separation) - White Gaussian noise, which results in some "background illumination" in the classic spectrogram, but creates some "wavery structure" in the reassigned spectrogram (which may be eliminated with a smarter algorithm one fine day...)

## Reassigned Spectrogram Options

<To Be Completed>

To switch the spectrogram display mode from "classic" to "reassigned", select (from SpecLab's main menu):

- Quick Settings .. Reassigned Spectrograms ... time- and frequency reassigned spectrogram

(selects the reassigned spectrogram display, but leaves all other settings unchanged) - Quick Settings .. Reassigned Spectrograms ... frequency- but not time-reassigned spectrogram

(was only intended for testing purposes, but it's interesting to see the effect of frequency-reassignment alone) - Quick Settings .. Reassigned Spectrograms ... Classic spectrogram display

(switches back to the "normal" spectrogram display, without changing anything else)

See also: Spectrum Lab's main index, display settings, Controls on the left side and on the bottom of the main window (separate documents).

## Principle

The classic spectrogram plots the magnitudes found in each frequency bin. It discards the phase information from the short-time fourier transform.

The reassigned spectrogram tries to "sharpen" the time/frequency display by using phase information from the complex short-time fourier transform (STFT).

For example, by comparing the phase between two neighbouring frequency bins (within the same STFT) it is possible to relocate the energy from that cell along the time(!) axis.

By comparing the phase in a frequency bin (between two neighbouring STFTs), it is possible to relocate the energy from that cell along the frequency(!) axis.

Details, and implementation notes can be found in the literature; some articles are listed further below.

To use the reassigned spectrogram effectively, one doesn't have to understand exactly how it works; but it's important to know the limitations.

<ToDo: Complete this>

## Limitations

### Separability

If two different signals are present in the same cell (overlapping in time, and/or in frequency), they cannot be separated.

The effect of non separable signals in a reassigned spectrogram is visible in image 1:

At the intersection between the vertical line (= the sharp falling edge of the sawtooth signal) and the slow frequency modulated sine wave, some pixels appear "scattered".

To a certain degree, such effects can be minimized by selecting the FFT size (number of points; and thus the FFT window length in seconds) so that:

- The FFT window time is shorter than the distance between two signals in the time domain;
- The width of one FFT frequency bin is small enough (and thus the FFT window time large enough) so that two signals never fall inside the same FFT frequency bin.

These important parameters are summarized on SpecLab's FFT configuration tab (from the main menu: "Options".."FFT Settings").

Here the FFT properties used in the test configuration:

In this example, two signals can only be separated (i.e. produce a "good" time/frequency reassigned spectrogram) if they are separated by at least 129 Hz (in frequency), or by at least 12 milliseconds (in time).

A higher frequency resolution decreases the time resolution, and vice versa. In this regard, the same limitations exist for the classic spectrogram and for the reassigned spectrogram. Carefully selecting the optimum FFT size is essential :

- Too large FFT size -> poor time separability, but good frequency separability
- Too small FFT size -> poor frequency separability, but good time separability .
- If there is only "one" signal (or a large separation between two signals, in time and/or frequency), the selection of the FFT size is not as critical as when there are two or more signals close to each other.
- It is advisable to use the "automatic scroll speed" for the spectrogram, and use an overlap (between two STFTs) of 50, 75, or 87.5 percent.

Then, the FFT size will also set the scroll speed (aka 'frame advance'), so there's not too much "smearing" along the time axis.

The preconfigured settings in Spectrum Lab's Quick Settings menu may help to find a good starting point for a certain application.

## Further reading

Since web links are so short-lived, do a websearch to find the following articles:

- [1] "Algorithms for computing the time-corrected instantaneous frequency (reassigned) spectrogram, with applications"

by Sean A. Fulop and Kelly R. Fitz, (c) Acoustical Society of America, 2006 . - [2] "A Unified Theory of Time-Frequency Reassignment" by the same authors

Last modified: 2009-06-12

back to top

The **method of reassignment** is a technique for sharpening a time-frequency representation by mapping the data to time-frequency coordinates that are nearer to the true region of support of the analyzed signal. The method has been independently introduced by several parties under various names, including *method of reassignment*, *remapping*, *time-frequency reassignment*, and *modified moving-window method*.^{[1]} In the case of the spectrogram or the short-time Fourier transform, the method of reassignment sharpens blurry time-frequency data by relocating the data according to local estimates of instantaneous frequency and group delay. This mapping to reassigned time-frequency coordinates is very precise for signals that are separable in time and frequency with respect to the analysis window.

## Introduction[edit]

Many signals of interest have a distribution of energy that varies in time and frequency. For example, any sound signal having a beginning or an end has an energy distribution that varies in time, and most sounds exhibit considerable variation in both time and frequency over their duration. Time-frequency representations are commonly used to analyze or characterize such signals. They map the one-dimensional time-domain signal into a two-dimensional function of time and frequency. A time-frequency representation describes the variation of spectral energy distribution over time, much as a musical score describes the variation of musical pitch over time.

In audio signal analysis, the spectrogram is the most commonly used time-frequency representation, probably because it is well understood, and immune to so-called "cross-terms" that sometimes make other time-frequency representations difficult to interpret. But the windowing operation required in spectrogram computation introduces an unsavory tradeoff between time resolution and frequency resolution, so spectrograms provide a time-frequency representation that is blurred in time, in frequency, or in both dimensions. The method of time-frequency reassignment is a technique for refocussing time-frequency data in a blurred representation like the spectrogram by mapping the data to time-frequency coordinates that are nearer to the true region of support of the analyzed signal.

## The spectrogram as a time-frequency representation[edit]

Main article: Spectrogram

One of the best-known time-frequency representations is the spectrogram, defined as the squared magnitude of the short-time Fourier transform. Though the short-time phase spectrum is known to contain important temporal information about the signal, this information is difficult to interpret, so typically, only the short-time magnitude spectrum is considered in short-time spectral analysis.

As a time-frequency representation, the spectrogram has relatively poor resolution. Time and frequency resolution are governed by the choice of analysis window and greater concentration in one domain is accompanied by greater smearing in the other.

A time-frequency representation having improved resolution, relative to the spectrogram, is the Wigner–Ville distribution, which may be interpreted as a short-time Fourier transform with a window function that is perfectly matched to the signal. The Wigner–Ville distribution is highly concentrated in time and frequency, but it is also highly nonlinear and non-local. Consequently, this distribution is very sensitive to noise, and generates cross-components that often mask the components of interest, making it difficult to extract useful information concerning the distribution of energy in multi-component signals.

Cohen's class of bilinear time-frequency representations is a class of "smoothed" Wigner–Ville distributions, employing a smoothing kernel that can reduce sensitivity of the distribution to noise and suppresses cross-components, at the expense of smearing the distribution in time and frequency. This smearing causes the distribution to be non-zero in regions where the true Wigner–Ville distribution shows no energy.

The spectrogram is a member of Cohen's class. It is a smoothed Wigner–Ville distribution with the smoothing kernel equal to the Wigner–Ville distribution of the analysis window. The method of reassignment smooths the Wigner–Ville distribution, but then refocuses the distribution back to the true regions of support of the signal components. The method has been shown to reduce time and frequency smearing of any member of Cohen's class ^{[2]} .^{[3]} In the case of the reassigned spectrogram, the short-time phase spectrum is used to correct the nominal time and frequency coordinates of the spectral data, and map it back nearer to the true regions of support of the analyzed signal.

## The method of reassignment[edit]

Pioneering work on the method of reassignment was published by Kodera, Gendrin, and de Villedary under the name of *Modified Moving Window Method*^{[4]} Their technique enhances the resolution in time and frequency of the classical Moving Window Method (equivalent to the spectrogram) by assigning to each data point a new time-frequency coordinate that better-reflects the distribution of energy in the analyzed signal.

In the classical moving window method, a time-domain signal, is decomposed into a set of coefficients, , based on a set of elementary signals, , defined

where is a (real-valued) lowpass kernel function, like the window function in the short-time Fourier transform. The coefficients in this decomposition are defined

where is the magnitude, and the phase, of , the Fourier transform of the signal shifted in time by and windowed by .

can be reconstructed from the moving window coefficients by

For signals having magnitude spectra, , whose time variation is slow relative to the phase variation, the maximum contribution to the reconstruction integral comes from the vicinity of the point satisfying the phase stationarity condition

or equivalently, around the point defined by

This phenomenon is known in such fields as optics as the principle of stationary phase, which states that for periodic or quasi-periodic signals, the variation of the Fourier phase spectrum not attributable to periodic oscillation is slow with respect to time in the vicinity of the frequency of oscillation, and in surrounding regions the variation is relatively rapid. Analogously, for impulsive signals, that are concentrated in time, the variation of the phase spectrum is slow with respect to frequency near the time of the impulse, and in surrounding regions the variation is relatively rapid.

In reconstruction, positive and negative contributions to the synthesized waveform cancel, due to destructive interference, in frequency regions of rapid phase variation. Only regions of slow phase variation (stationary phase) will contribute significantly to the reconstruction, and the maximum contribution (center of gravity) occurs at the point where the phase is changing most slowly with respect to time and frequency.

The time-frequency coordinates thus computed are equal to the local group delay, and local instantaneous frequency, and are computed from the phase of the short-time Fourier transform, which is normally ignored when constructing the spectrogram. These quantities are *local* in the sense that they represent a windowed and filtered signal that is localized in time and frequency, and are not global properties of the signal under analysis.

The modified moving window method, or method of reassignment, changes (reassigns) the point of attribution of to this point of maximum contribution , rather than to the point at which it is computed. This point is sometimes called the *center of gravity* of the distribution, by way of analogy to a mass distribution. This analogy is a useful reminder that the attribution of spectral energy to the center of gravity of its distribution only makes sense when there is energy to attribute, so the method of reassignment has no meaning at points where the spectrogram is zero-valued.

## Efficient computation of reassigned times and frequencies[edit]

In digital signal processing, it is most common to sample the time and frequency domains. The discrete Fourier transform is used to compute samples of the Fourier transform from samples of a time domain signal. The reassignment operations proposed by Kodera et al. cannot be applied directly to the discrete short-time Fourier transform data, because partial derivatives cannot be computed directly on data that is discrete in time and frequency, and it has been suggested that this difficulty has been the primary barrier to wider use of the method of reassignment.

It is possible to approximate the partial derivatives using finite differences. For example, the phase spectrum can be evaluated at two nearby times, and the partial derivative with respect to time be approximated as the difference between the two values divided by the time difference, as in

For sufficiently small values of and and provided that the phase difference is appropriately "unwrapped", this finite-difference method yields good approximations to the partial derivatives of phase, because in regions of the spectrum in which the evolution of the phase is dominated by rotation due to sinusoidal oscillation of a single, nearby component, the phase is a linear function.

Independently of Kodera *et al.*, Nelson arrived at a similar method for improving the time-frequency precision of short-time spectral data from partial derivatives of the short-time phase spectrum.^{[5]} It is easily shown that Nelson's *cross spectral surfaces* compute an approximation of the derivatives that is equivalent to the finite differences method.

Auger and Flandrin showed that the method of reassignment, proposed in the context of the spectrogram by Kodera et al., could be extended to any member of Cohen's class of time-frequency representations by generalizing the reassignment operations to

where is the Wigner–Ville distribution of , and is the kernel function that defines the distribution. They further described an efficient method for computing the times and frequencies for the reassigned spectrogram efficiently and accurately without explicitly computing the partial derivatives of phase.^{[2]}

In the case of the spectrogram, the reassignment operations can be computed by

where is the short-time Fourier transform computed using an analysis window is the short-time Fourier transform computed using a time-weighted analysis window and is the short-time Fourier transform computed using a time-derivative analysis window .

Using the auxiliary window functions and , the reassignment operations can be computed at any time-frequency coordinate from an algebraic combination of three Fourier transforms evaluated at