Whilst the video is playing, click on a line in the transcript to play the video from that point. So I'm with a remainder of the class, which is all about feature engineering, and we'll start with the magnitude spectrum and the filter bank features that we've seen before in the last module. Quick recap. We extract frames from a speech waveform. This is in the time domain. We extract short-term analysis frames to avoid discontinuities at the edges. We apply the discrete Fourier transform and we get the magnitude spectrum. This is written on a logarithmic scale, so this is the log magnitude spectrum. And from that, we extract filter bank features. Filter bank features are found from a single frame of speech in the magnitude spectrum domain. We divide it into bands, spaced on a mel scale, and sum the energy in each band and write that into an element of the feature vector. We typically use triangular shape filters by spacing their centers and their widths on the mel scale. They get further and further apart and wider and wider as we go up frequency in hertz. And the energy in each of those, for example, the energy summed in this one, is written into the first element of our feature vector, of a multidimensional feature vector. Now let's make it really clear that the features in this feature vector will exhibit a lot of covariance. They are highly correlated with each other. And that will become very obvious when we look at this animation. So remember that each band of frequencies is going into the elements of this feature vector, extracted with these triangular filters. So when I play the animation, just look at energy in adjacent frequency bands. Clearly, the energy in this band and the energy in this band are going up and down together. See that? When this one goes up, this one tends to go up. And when this one goes down, this one tends to go down. They're highly correlated. They co-vary because they're adjacent. And the spectral envelope is a smooth thing. And so these features are highly correlated. And if we wanted to model this feature vector with a multivariate Gaussian, it would be important to have a full covariance matrix to capture that. So the filter bank energies themselves are perfectly good features, unless we want to model them with a diagonal covariance Gaussian, which is what we've decided to do. So we're going to do some feature engineering to get around that problem. So we're going to build up to some features called Mel Frequency Cepstral Coefficients. And MFCCs, as they're usually called, take inspiration from a number of different directions. And one strong way of motivating them is to actually go back and think about speech production again and to remember what we learned a little while ago about convolution. Knowing what we know about convolution, we're going to derive a form of analysis called Cepstral analysis. Cepstral is an anagram of spectral. It's a transform of spectral. So here's a recap that convolution in the time domain, convolution of waveforms, is equivalent to addition in the log magnitude spectrum domain. So just put aside the idea of the filter bank for a moment and let's go right back to the time domain and start again. This is our idealized source. This is the impulse response of the vocal track filter. And if we convolve those, that's what the star means, we'll get our speech signal in the time domain. And that's equivalent to transforming each of those into the spectral domain and plotting on a log scale. And then we see that their log magnitude spectra add together. So this becomes addition in the log magnitude spectrum domain. So convolution is a complicated operation and we might imagine perfectly reasonably that deconvolution is very hard. In other words, given that to go backwards and decompose it into source and filter in the time domain is hard, but we could imagine that undoing an addition is rather easier. And that's exactly what we're about to do. How do we get from the time domain to the frequency domain? We'll use the Fourier transform. The Fourier transform is just a series expansion. So to get from this time domain signal to this frequency domain signal, we did a transform that's a series expansion. And that series expansion had the effect of turning this axis from time in units of seconds to this axis in frequency, which has units of what? Hertz. But Hertz are just one over seconds. So the series expansion has the effect of changing the axes to be one over the original axis. So we start with something in the time domain, we end up with something in the one over time domain or frequency domain. So that's a picture of speech production. But we don't get to see that when we're doing speech recognition. All we get is a speech signal from which we can compute the log magnitude spectrum. What would I like to get for doing automatic speech recognition? Well, I've said that fundamental frequency is not of interest. I would like the vocal tract frequency response. That's the most useful feature for doing speech recognition. But what I start with is this. So can I undo the summation? So that's how speech is produced, but we don't have access to that. We would like to do this. We would like to start from, we can easily compute with a Fourier transform from an analysis frame of speech. And we would like to decompose that into a sum of two parts, filter plus source. And then for speech recognition, we'll just discard the source. How might you solve this equation given only the thing on the left? Well, one obvious option is to use a source filter model. We could use a filter that's defined by its difference equation, and we could solve for the coefficients of that difference equation. And that will give us this part. And then we could just subtract that from the original signal and whatever's left must be this part here, which we might then call a remainder or more formally call it the residual. And we'll assume that was the source. So that will be an explicit way of decomposing into source and filter, and we'd get both the source and the filter. But actually we don't care about the source for speech recognition. We just want the filter so we can do something a little bit simpler. Fitting an explicit source filter model involves making very strong assumptions about the form of the filter. For example, if it's a resonant filter, it's all pole, and that the difference equation has a very particular form with a particular number of coefficients. We might not want to make such a strong assumption. And solving for that difference equation, solving for the coefficients given a frame of speech waveform can be error prone. And it's actually not something we cover in this course. So I want to solve this apparently difficult to solve equation where we know the thing on the left, and we want to turn it into some of two things on the right. These two things have quite different looking properties. With respect to this axis here, which is frequency, this one's quite slowly varying, smooth and slowly varying. It's a slow function of frequency. With respect to the frequency axis, this one here, it's quite rapidly moving. It changes rapidly with respect to frequency. So we would like to decompose this into the slowly varying part and the rapidly varying part. And I mean slowly and rapidly varying with respect to this axis, the frequency axis. So I can't directly do that into these two parts, but I can write something more general down like this. I can say that a log magnitude spectrum of an analysis frame of speech equals something plus something, plus something, plus something, and so on, where we start off with very slowly varying parts and then slightly quicker varying all the way up to eventually very rapidly varying parts. Does that look familiar? I hope so. That's a series expansion, not unlike Fourier analysis. So it's a transform. We're going to transform the log magnitude spectrum into a summation of basis functions weighted by coefficients. Well, we could use the same basis functions as Fourier analysis, in other words, sinusoids with magnitude and phase or any other suitable set of basis functions. The only important thing is that the basis functions have to be orthogonal. So they have to be a series of orthogonal functions that don't correlate with each other. So go and revise the series expansion video if you need to remember what that is. In this particular case, we're doing a series expansion of the log magnitude spectrum. The most popular choice of basis functions is actually a series of cosines where we just need the magnitude of each. There's no phase, they're just exactly cosines. That suits the particular properties of the log magnitude spectrum. So we're going to write down this. This part here equals a sum of some constant function times some coefficient, a weight, plus some amount of this function, that's the lowest frequency cosine we can fit in there, times some weight, plus some amount of the next one, plus some amount of the next one, and so on for as long as we like. There's a set of orthogonal basis functions that's a cosine series. It starts with this one, which is just the offset, the zero frequency component, if you like, and then works its way up through the series. And so we'll be characterizing the log magnitude spectrum by these coefficients. This is a cosine transform. Lots and lots of textbooks give you a really, really useless and unhelpful analogy to try and understand what's happening here. They'll say the following. They say, let's pretend this is time and then do the Fourier transform, but we don't need to do that. A series expansion is a series expansion. There's nothing here that requires this label on this axis to be frequency, time, or anything else. You don't need to pretend that's a time axis. You don't need to pretend this is a Fourier transform. This is just a series expansion. You've got some complicated function and you're expressing it as a sum of simple functions weighted by coefficients. It's those coefficients that characterize this particular function that we're expanding. Now how does this help us solve what we wanted to solve, which is just to write this thing, this magnitude spectrum, as just a sum of two parts, a slowly varying part with respect to frequency that we'll say is the filter, and a rapidly varying part we'll say is the source because here we've got some long series of parts. Well we'll just work our way up through the series and at some point say that's as rapidly varying as we ever see the filter with respect to frequency and we'll stop and draw a line and everything after that we'll say is not the filter. So we'll just count up through these series and at some point we'll stop and we'll say the slow ones are the filter and the rapid ones are the source. So all we've got to do in fact now is decide where to draw the line and there's a very common choice of value there and that's to keep the first 12 basis functions. Counting this one as number one. This is a special one. It's just the energy and we count that as zero. So that's the first basis function, the second, the third, and we go up to the twelfth. And in other descriptions of Cepstral analysis, particularly of the form used to extract MFCCs, you might see choices other than the cosine basis functions. You could use the Fourier series for example. That's a detail that's not important. The important thing is conceptually this is a series expansion into a series of orthogonal basis functions. Exactly what functions you expand it into is less important. We won't get into an argument about that. We'll just say cosine series. What is the output of that series expansion going to look like? Well, just like any other series expansion such as the Fourier transform, we'll plot those coefficients on an axis. This is frequency in Hertz. Frequency is the same as one over time. So that series expansion gives you a new axis, which is one over the previous axis. So one over one over time is time. So actually we're going to have something that's got a time axis, time scale. This is going to be the size of the coefficient of each of the basis functions. So we're going to go for 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 basis functions. And at that point we'll say these guys belong to the filter and then everything else to the right we're actually going to discard because it belongs to the source. But let's see what it might look like. Well, we just have some values. This thing is going to be hard to interpret, going to be the coefficients of the cosine series. But what we'll find if we kept going well past 12, at some point there would be a coefficient at some high time where we get some spike again. Let's think about what that means on the magnitude spectrum. So this is the cosine series expansion. These lower coefficients here, now this is called, this is time. But because this is a transform from frequency onwards, we don't typically label it with time. We label it with an anagram of frequency and people use the word queferency. Don't ask me why. It's the units are seconds. So this low queferency coefficient is the slowly moving part. One of these perhaps here is some faster moving part, up here some faster moving part, perhaps this one. And eventually this one will be the one that moves at this rate. This is a cosine function that happens to just snap onto the harmonics and it will just match the harmonics. So this one here is going to be the fundamental period. Because this matches the harmonics at F0. For our purposes, we're going to stop here, going to throw away all of these as being the fine detail and retain the first 12, 1 to 12. So this truncation is what separates source and filter and specifically it just retains the filter and discards the source. Truncation of a series expansion is actually a very well principled way to smooth a function. So essentially we just smooth the function on the left to remove the fine detail, the harmonics, and retain the detail up to some certain scale and our scale is up to 12 coefficients.
Cepstral Analysis, mel-scale Filterbanks, MFCCs
We now start thinking about what a good representation of the acoustic signal should be, motivating the use of Mel-Frequency Cepstral Coefficients (MFCCs).
Log in if you want to mark this as completed
|
| ||||||||||||||||||||||||||||||||||||||||||