Total video to watch in this section: 44 minutes (divided into 4 parts for easier viewing)
Here are the slides for these videos.
Assumptions & references made in these videos
The opening slide in the video “Intro to Generative Models” says “Module 7” – this is now Module 8.
The video “Intro to Generative Models” assumes familiarity with discrete and continuous (random) variables, and Bayes rule. The foundation material on probability will be enough for you to understand this video, so go over that first if you need to. We will then consolidate these topics in the lecture for this module.
The last slide in the video “From MFCCs, towards a generative model using HMMs” states that HMMs are in Module 8: that should now read Module 9. It states that estimating the parameters of an HMM is in Module 9: that should now read Module 10.
Whilst the video is playing, click on a line in the transcript to play the video from that point. This module has two parts. In the first part, we're going to start building the Hidden Markov Model, but only get as far as deciding to use Gaussian probability density functions as the heart of the model. And that will make us pause for thought and think about the features that we need to use with Gaussian probability density functions. So in this first part, I'm going to introduce a rather challenging concept, and that's the idea of a generative model. And we'll choose the Gaussian as our generative model for the features we have for speech recognition. The main conclusion from that part will be that we don't want to model covariance. That is, we're going to have a multivariate Gaussian. That's a Gaussian in a feature space that's multidimensional, the feature vectors extracted from each frame of speech. And we would like to assume that there's no covariance between any pair of elements in that feature vector. For the features that we've seen so far, filter bank features, that's not going to be true. So those features are not going to be directly suitable for modeling with a Gaussian that has no covariance parameters. We often call such a Gaussian diagonal covariance because only the diagonal entries of its covariance matrix have non-zero values. So we'll conclude that we want diagonal covariance Gaussians. We'll realize that filter bank features are not suitable for that. And we need to do what's called feature engineering to those features, some further steps of processing to remove that covariance. And that will result in the final set of features that we're going to use in this course, which are called Mel Frequency Cepstral Coefficients. And to get to them, we'll need to understand what the Cepstral is. So let's orient ourselves in the course as a whole. We're on a journey towards HMMs. We've talked about pattern matching, and we've seen that already in the form of dynamic time warping. There was no probability there. We were comparing a template, an exemplar, with an unknown and measuring distance. And the distance measure was a Euclidean distance for one feature vector in the exemplar compared to one feature vector in the unknown. The Euclidean distance is really what's at the core of the Gaussian probability density function. Our conclusion from doing simple pattern matching was that it fails to capture the variability of speech, the natural variability around the mean. A single exemplar is like having just the mean, but we want also the variability around that. That's why we're moving away from exemplars to probabilistic models. So to bridge between pattern matching and probabilistic generative modelling, we're going to have to think again about features, because there's going to be an interaction between the choice of features and choice of model. In particular, we're going to choose a model that makes assumptions about the features, and then we'll need to do things to the features to make those assumptions as true as possible. We've already talked enough about probability and about the Gaussian probability density function in last week's tutorials. We've also gone over why we don't want to model covariance. We also looked at human hearing, and we decided to take a little bit of inspiration from how the cochlear operates. The core reason we don't want to model covariance is it massively increases the number of parameters in the Gaussian, and that would require a lot more training data to get good estimates of all those parameters. And we're not going to explicitly model human hearing. We're going to use it as a kind of inspiration for some of the things we do in feature extraction. So without repeating too much material from last week, let's have a very quick recap of some concepts that we're going to need. We're modelling features such as the output of a filter in the filter bank, and that's a continuously valued thing. So we're going to need to deal with continuous variables, but also we're going to need to deal with discrete variables. At some point, we're going to decide speech as a sequence of words. Words are discrete. We can count the number of types, and maybe we're going to decompose the word into a sequence of phonemes, which is also a countable thing, a discrete thing. So we're going to need to deal with both discrete and continuous variables, and these are going to be random variables. They're going to take a distribution over a set of possible values or over a range of possible values. An example of a discrete variable then would be the word, and in the assignment, that will be one of 10 possible values. An example of a continuous variable will be the output from one of the filters in the filter bank, or indeed the vector of all the outputs stacked together. That's just a vector space, and that is also a continuous variable in a multidimensional space. And we'd need different types of probability distribution to represent them. But apart from that, everything else that we've learned about probability theory, and in particular, this idea of joint distributions, conditional distributions, and the all-important Bayes formula, the theorem that tells us how to connect all of these things, applies to both discrete and continuous variables. It's true for all of them. And in fact, eventually, we're going to write down Bayes formula with a mixture of discrete and continuous variables in it, words and acoustic features. But that won't be a problem, because there's nothing in all of what we learned about Bayes that tells us it only works for one or the other. It works for both discrete and continuous variables. So I'm not going to go over that material again. I'm going to assume that we know that. So let's proceed. We can use probability distributions to describe data. That's kind of a summary of the data. And when we do that, we're making an assumption about how those data are distributed. For example, if we describe data using a Gaussian distribution, we're assuming that they have a normal distribution. And we might even make some stronger assumptions that they have no covariance. But before choosing the Gaussian, let's stay a little abstract for a while. We're not going to choose any particular model. We're going to introduce a concept, a difficult concept. This will be difficult to understand the first time you encounter it. It's a different way of thinking about what we do when we have a model. What is a model? What is a model for? What can it do? We're going to actually build the simplest possible form of model. So a generative model is actually as simple as it gets, because a generative model can only do one thing. It can generate examples from its own class. So there's a difficult concept to learn, but it's necessary because it allows us to build the simplest form of model, the generative model. So here's the conceptual leap. Here are three models of three different classes, A, B, and C. We're keeping this abstract. I'm not saying anything about what form of model we're using. So these are black boxes. And the only thing that a model can do, for example, model A here is emit observations. Let's make it do that. The observations here are balls and they just have one feature and that's their colour. So that just emitted a blue ball. And that's all this model can do. We can randomly emit observations. We can generate from the model. So let's generate. There's another one. So we're generating observations. The observations are data points and so on. And we keep generating. So this is a random generator and it generates from its distribution. So inside model A is a distribution over all the different colours of balls that are possible. And it randomly chooses from that according to that distribution and emits an observation. So keeping things abstract and not making any statements about what form the model has, it's not got any probability distribution. What are we going to say about model A? Well, we could imagine that it essentially contains lots and lots of data from class A like that. And we might have other models too for other classes. And we could imagine for now that they just contain a lot of examples of their classes ready to be generated, ready to be emitted. That's model B and model C. And you can see that it seems likely that model A was going to emit a different distribution of values, of colours, than model B. And model B is going to emit a different distribution to model C. So let's see what we could do with something like that. So we've got three generative models. All they can do is emit observations. Already, and keeping things very abstract without saying anything about what type of model we're using, let's see already that this simple set of three models, A, B and C, one for each of the classes, can do classification. Even though all the models can do is emit observations, they don't make decisions, but we can use them to make decisions, to classify. So let's see how a set of generative models can together be used to classify new samples, test samples. So here comes a data point. There it is. It's a ball and its feature is green. And I would like to answer the question, which of these models is most likely to have generated it? And if I can find the most likely model, I'll label this data point with the class of that model. So we're keeping the models abstract. We're pretending that they just contain big bags of balls. So that's what our models look like inside. So we try and generate from the model and we make the model generate this observation. One way of thinking about that, if you want to think in a frequentist way, a counting way, is to emit a batch of observations from each model and count how many of them were green. That's the frequentist view. The Bayesian view is just to say, what's the fraction, what's the probability of emissions of observations from a model that are green? And we can look at model A and it's pretty obvious that it's never going to emit a green ball. So the probability of this green ball having been emitted from model A is a probability of zero. I'll write it with one decimal place for precision. Look at model B. It's also clear just by looking that model B never emits green balls either. And the probability of that emitting a green is 0.0. Now let's look at model C. You can see that in model C, it emits blue balls and yellow balls and green balls, let's say with approximately equal proportions. So there's about a 0.3 probability of model C emitting a green ball. Clearly, this is the largest probability of the three. We'll classify this as being of class C. Notice that each model could compute only its own probability. It didn't need to know anything about the other models to do that. And each model on its own made no decisions. It merely placed a probability on this observed green ball. We then made the decision by comparing the probabilities. So the classifier is the comparison of the probabilities of these models emitting this observation. That's what makes these models simple, that they don't need to know about each other. They're all quite separated. And in particular, when we learn these models from data, we'll be able to learn them from just data labeled with their class. They won't need to see counter examples. They won't need to see examples from other classes. And that's the simplicity of the generative model. Other models are available, models that might be called discriminative models, that know about where the boundary is between, for example, class A and class B, and can draw that decision boundary and directly make the decision as to whether an observation falls on the A side of that line or the B side of that line, of that classification boundary. But these generative models do not do that. We have to form a classifier by comparing their probabilities. Let's classify another observation. So along comes an observation, and our job is to label it as being most likely class A or class B or class C. We just do the same operation again. We ask model A, model A, what is the probability that you generate a red ball? We can see just visually for model A, maybe it emits red balls about two thirds of the time. So model A says the probability of me emitting a red ball is about 0.6. Ask model B, model B, what's the probability that you emitted a red ball? Well, we should look, and model B emits red balls and yellow balls and blue balls. They're in about equal proportion. So it says I can emit red balls, and I can do it about a third of the time. Model C, what's the probability that you emitted a red ball? Well, it never emits a red ball. Zero. How do we make the classification decision? We compare these three probabilities, we pick the largest, and we label this test sample with the class of the highest probability model, which is an A. So this is labeled as A. So we've done classification. So you get the idea. Along comes a sample. You ask each of the models in turn, what's the probability that this is a sample from you, from your model, from your internal distribution, whatever that might be. We compute those three probabilities and pick the highest and label the test sample with that class. So as simple as generative models are, they can do classification simply by comparing probabilities.
Whilst the video is playing, click on a line in the transcript to play the video from that point. Let's move on from discrete things and colored balls to continuous values because that's what our features for speech recognition are They're extracted from frames of speech And so far where we've got with that is filter bank features A vector of numbers Each number is the energy in a filter in a band of frequencies for that frame of speech So we need a model of continuous values And the model we're going to choose is the Gaussian So I'm going to assume you know about Gaussians from last week's tutorial But let's just have a very quick reminder Let's do this in two dimensions So I'll just pick two of the filters in the filter bank and draw that two-dimensional space Perhaps I'll pick the third filter and the fourth filter in the filter bank And each of the points I'm going to draw is the pair of filter bank energies It's a little feature vector So each point is a little two-dimensional feature vector containing the energy in the third filter and the energy in the fourth filter So lots of data points If I would like to describe the distribution of this data with a Gaussian It's going to be a multivariate Gaussian This means going to be a vector of two dimensions And its covariance matrix is going to be a two by two matrix I'm going to have here a full covariance matrix Which means I could draw a Gaussian that is this shape on the data We've made the assumption here that the data are distributed normally And so that this parametric probability density function is a good representation of this data So I can use the Gaussian to describe data But how would we use the Gaussian as a generative model? Let's do that But let's just do it in one dimension to make things a bit easier to draw So here I've got my three models again And by some means yet to be determined I've learned these models They've come from somewhere And these models are now Gaussians So this is really what the models look like Model A is this Gaussian It has a particular mean and a particular standard deviation Along comes an observation So these are univariate Gaussians Our feature vectors are one-dimensional feature vectors So along comes a one-dimensional feature vector It's just a number And the question is Which of these models is most likely to have generated that number? Here's the number 2.1 Now remember that a Gaussian can't compute a probability That would involve integrating the area between two values So for 2.1 all we can say is what's the probability density at 2.1 So off we go 2.1 This value 2.1 This value 2.1 This value Compare those three Clearly this one is the highest And so we'll say this is an A And that's how we'd use these three Gaussians as generative models We'd ask each of them in turn Can you generate the value 2.1? Now for a Gaussian the answer is always yes Because all values have non-zero probability So of course we can generate a 2.1 What's the probability density at 2.1? We just read that off the curve Because it's a parametric distribution And compare those three probability densities So we've done classification with a Gaussian Let's just draw the three models on top of each other On top of each other to make it even clearer What's the probability of 2.1 being an A or a B or a C?
Whilst the video is playing, click on a line in the transcript to play the video from that point. So I'm with a remainder of the class, which is all about feature engineering, and we'll start with the magnitude spectrum and the filter bank features that we've seen before in the last module. Quick recap. We extract frames from a speech waveform. This is in the time domain. We extract short-term analysis frames to avoid discontinuities at the edges. We apply the discrete Fourier transform and we get the magnitude spectrum. This is written on a logarithmic scale, so this is the log magnitude spectrum. And from that, we extract filter bank features. Filter bank features are found from a single frame of speech in the magnitude spectrum domain. We divide it into bands, spaced on a mel scale, and sum the energy in each band and write that into an element of the feature vector. We typically use triangular shape filters by spacing their centers and their widths on the mel scale. They get further and further apart and wider and wider as we go up frequency in hertz. And the energy in each of those, for example, the energy summed in this one, is written into the first element of our feature vector, of a multidimensional feature vector. Now let's make it really clear that the features in this feature vector will exhibit a lot of covariance. They are highly correlated with each other. And that will become very obvious when we look at this animation. So remember that each band of frequencies is going into the elements of this feature vector, extracted with these triangular filters. So when I play the animation, just look at energy in adjacent frequency bands. Clearly, the energy in this band and the energy in this band are going up and down together. See that? When this one goes up, this one tends to go up. And when this one goes down, this one tends to go down. They're highly correlated. They co-vary because they're adjacent. And the spectral envelope is a smooth thing. And so these features are highly correlated. And if we wanted to model this feature vector with a multivariate Gaussian, it would be important to have a full covariance matrix to capture that. So the filter bank energies themselves are perfectly good features, unless we want to model them with a diagonal covariance Gaussian, which is what we've decided to do. So we're going to do some feature engineering to get around that problem. So we're going to build up to some features called Mel Frequency Cepstral Coefficients. And MFCCs, as they're usually called, take inspiration from a number of different directions. And one strong way of motivating them is to actually go back and think about speech production again and to remember what we learned a little while ago about convolution. Knowing what we know about convolution, we're going to derive a form of analysis called Cepstral analysis. Cepstral is an anagram of spectral. It's a transform of spectral. So here's a recap that convolution in the time domain, convolution of waveforms, is equivalent to addition in the log magnitude spectrum domain. So just put aside the idea of the filter bank for a moment and let's go right back to the time domain and start again. This is our idealized source. This is the impulse response of the vocal track filter. And if we convolve those, that's what the star means, we'll get our speech signal in the time domain. And that's equivalent to transforming each of those into the spectral domain and plotting on a log scale. And then we see that their log magnitude spectra add together. So this becomes addition in the log magnitude spectrum domain. So convolution is a complicated operation and we might imagine perfectly reasonably that deconvolution is very hard. In other words, given that to go backwards and decompose it into source and filter in the time domain is hard, but we could imagine that undoing an addition is rather easier. And that's exactly what we're about to do. How do we get from the time domain to the frequency domain? We'll use the Fourier transform. The Fourier transform is just a series expansion. So to get from this time domain signal to this frequency domain signal, we did a transform that's a series expansion. And that series expansion had the effect of turning this axis from time in units of seconds to this axis in frequency, which has units of what? Hertz. But Hertz are just one over seconds. So the series expansion has the effect of changing the axes to be one over the original axis. So we start with something in the time domain, we end up with something in the one over time domain or frequency domain. So that's a picture of speech production. But we don't get to see that when we're doing speech recognition. All we get is a speech signal from which we can compute the log magnitude spectrum. What would I like to get for doing automatic speech recognition? Well, I've said that fundamental frequency is not of interest. I would like the vocal tract frequency response. That's the most useful feature for doing speech recognition. But what I start with is this. So can I undo the summation? So that's how speech is produced, but we don't have access to that. We would like to do this. We would like to start from, we can easily compute with a Fourier transform from an analysis frame of speech. And we would like to decompose that into a sum of two parts, filter plus source. And then for speech recognition, we'll just discard the source. How might you solve this equation given only the thing on the left? Well, one obvious option is to use a source filter model. We could use a filter that's defined by its difference equation, and we could solve for the coefficients of that difference equation. And that will give us this part. And then we could just subtract that from the original signal and whatever's left must be this part here, which we might then call a remainder or more formally call it the residual. And we'll assume that was the source. So that will be an explicit way of decomposing into source and filter, and we'd get both the source and the filter. But actually we don't care about the source for speech recognition. We just want the filter so we can do something a little bit simpler. Fitting an explicit source filter model involves making very strong assumptions about the form of the filter. For example, if it's a resonant filter, it's all pole, and that the difference equation has a very particular form with a particular number of coefficients. We might not want to make such a strong assumption. And solving for that difference equation, solving for the coefficients given a frame of speech waveform can be error prone. And it's actually not something we cover in this course. So I want to solve this apparently difficult to solve equation where we know the thing on the left, and we want to turn it into some of two things on the right. These two things have quite different looking properties. With respect to this axis here, which is frequency, this one's quite slowly varying, smooth and slowly varying. It's a slow function of frequency. With respect to the frequency axis, this one here, it's quite rapidly moving. It changes rapidly with respect to frequency. So we would like to decompose this into the slowly varying part and the rapidly varying part. And I mean slowly and rapidly varying with respect to this axis, the frequency axis. So I can't directly do that into these two parts, but I can write something more general down like this. I can say that a log magnitude spectrum of an analysis frame of speech equals something plus something, plus something, plus something, and so on, where we start off with very slowly varying parts and then slightly quicker varying all the way up to eventually very rapidly varying parts. Does that look familiar? I hope so. That's a series expansion, not unlike Fourier analysis. So it's a transform. We're going to transform the log magnitude spectrum into a summation of basis functions weighted by coefficients. Well, we could use the same basis functions as Fourier analysis, in other words, sinusoids with magnitude and phase or any other suitable set of basis functions. The only important thing is that the basis functions have to be orthogonal. So they have to be a series of orthogonal functions that don't correlate with each other. So go and revise the series expansion video if you need to remember what that is. In this particular case, we're doing a series expansion of the log magnitude spectrum. The most popular choice of basis functions is actually a series of cosines where we just need the magnitude of each. There's no phase, they're just exactly cosines. That suits the particular properties of the log magnitude spectrum. So we're going to write down this. This part here equals a sum of some constant function times some coefficient, a weight, plus some amount of this function, that's the lowest frequency cosine we can fit in there, times some weight, plus some amount of the next one, plus some amount of the next one, and so on for as long as we like. There's a set of orthogonal basis functions that's a cosine series. It starts with this one, which is just the offset, the zero frequency component, if you like, and then works its way up through the series. And so we'll be characterizing the log magnitude spectrum by these coefficients. This is a cosine transform. Lots and lots of textbooks give you a really, really useless and unhelpful analogy to try and understand what's happening here. They'll say the following. They say, let's pretend this is time and then do the Fourier transform, but we don't need to do that. A series expansion is a series expansion. There's nothing here that requires this label on this axis to be frequency, time, or anything else. You don't need to pretend that's a time axis. You don't need to pretend this is a Fourier transform. This is just a series expansion. You've got some complicated function and you're expressing it as a sum of simple functions weighted by coefficients. It's those coefficients that characterize this particular function that we're expanding. Now how does this help us solve what we wanted to solve, which is just to write this thing, this magnitude spectrum, as just a sum of two parts, a slowly varying part with respect to frequency that we'll say is the filter, and a rapidly varying part we'll say is the source because here we've got some long series of parts. Well we'll just work our way up through the series and at some point say that's as rapidly varying as we ever see the filter with respect to frequency and we'll stop and draw a line and everything after that we'll say is not the filter. So we'll just count up through these series and at some point we'll stop and we'll say the slow ones are the filter and the rapid ones are the source. So all we've got to do in fact now is decide where to draw the line and there's a very common choice of value there and that's to keep the first 12 basis functions. Counting this one as number one. This is a special one. It's just the energy and we count that as zero. So that's the first basis function, the second, the third, and we go up to the twelfth. And in other descriptions of Cepstral analysis, particularly of the form used to extract MFCCs, you might see choices other than the cosine basis functions. You could use the Fourier series for example. That's a detail that's not important. The important thing is conceptually this is a series expansion into a series of orthogonal basis functions. Exactly what functions you expand it into is less important. We won't get into an argument about that. We'll just say cosine series. What is the output of that series expansion going to look like? Well, just like any other series expansion such as the Fourier transform, we'll plot those coefficients on an axis. This is frequency in Hertz. Frequency is the same as one over time. So that series expansion gives you a new axis, which is one over the previous axis. So one over one over time is time. So actually we're going to have something that's got a time axis, time scale. This is going to be the size of the coefficient of each of the basis functions. So we're going to go for 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 basis functions. And at that point we'll say these guys belong to the filter and then everything else to the right we're actually going to discard because it belongs to the source. But let's see what it might look like. Well, we just have some values. This thing is going to be hard to interpret, going to be the coefficients of the cosine series. But what we'll find if we kept going well past 12, at some point there would be a coefficient at some high time where we get some spike again. Let's think about what that means on the magnitude spectrum. So this is the cosine series expansion. These lower coefficients here, now this is called, this is time. But because this is a transform from frequency onwards, we don't typically label it with time. We label it with an anagram of frequency and people use the word queferency. Don't ask me why. It's the units are seconds. So this low queferency coefficient is the slowly moving part. One of these perhaps here is some faster moving part, up here some faster moving part, perhaps this one. And eventually this one will be the one that moves at this rate. This is a cosine function that happens to just snap onto the harmonics and it will just match the harmonics. So this one here is going to be the fundamental period. Because this matches the harmonics at F0. For our purposes, we're going to stop here, going to throw away all of these as being the fine detail and retain the first 12, 1 to 12. So this truncation is what separates source and filter and specifically it just retains the filter and discards the source. Truncation of a series expansion is actually a very well principled way to smooth a function. So essentially we just smooth the function on the left to remove the fine detail, the harmonics, and retain the detail up to some certain scale and our scale is up to 12 coefficients.
Whilst the video is playing, click on a line in the transcript to play the video from that point. Now what we just saw there was the true cepstrum, which we got from the original log magnitude spectrum That's going to inspire one of the processing steps that's coming now in the feature extraction pipeline for Mel Frequency Cepstral Coefficients So let's get back on track with that. We've got the frequency domain, the magnitude spectrum We've applied a filter bank to that primarily to warp the scale to the mel scale But we can also conveniently choose filter bandwidths that smooth away much of the evidence of f0 and We'll take the log of the output of those filters. So when we implement this Normally what we have here is the linear magnitude spectrogram Apply the triangular filters and then take the log of their outputs You can plot those against frequency and draw something that's essentially the spectral envelope of speech And we then take inspiration from the cepstral transform and do as a series expansion of that we'll use Cosine basis functions and that's called cepstral analysis and that will give us the cepstrum and then we decide how many coefficients we'd like to retain So we'll truncate that series Typically a coefficient number 12 and that will give us these wonderful coefficients called Mel Frequency Cepstral Coefficients In other descriptions of MFCCs you might see things other than the log here some other compressive function Such as the cube root. That's a detail that's not conceptually important. What's important is that it's compressive The log is the most well motivated because it comes from the true cepstrum which we get from the log magnitude spectrum It's the thing that turns multiplication into addition This truncation here serves several purposes It could be that our filter bank didn't entirely smooth away all the evidence of f0 and So truncation will discard any remaining detail in their outputs So we'll get a very smooth spectral envelope by truncating the series So it removes any remaining evidence of the source just in case the filter bank didn't do it completely. That's number one The second thing it does I'm going to explain in a moment We've alluded to is that by expanding into a series of orthogonal basis functions We find a set of coefficients that don't exhibit much covariance with each other so we've removed covariance through this series expansion and third and Just as interesting We've got a place where we can control how many features we get 12 is the most common choice But you could vary that and we get the 12 most important features The ones that the detail in the spectral envelope up to a certain fineness of detail There's a well motivated way of controlling the dimensionality of our feature vector. So did we remove covariance? Well, we could answer that question theoretically, which we'll try and do now We could also answer that question empirically by experiment. We could do experiments where we use Gaussians Multivariate Gaussians with diagonal covariance and we use filter bank features and we compare how good that system is with MFCCs and in general, we'll find that MFCCs are better The theoretical argument is that we expanded into a series of orthogonal basis functions Which have no correlation with each other they're independent These Are uncorrelated with each other and that's the theoretical reason why a series expansion gives you a set of coefficients Which don't have covariance or at least have a lot less covariance than the original filter bank coefficients So finally, we've done cepstral analysis and we've got MFCCs MFCCs are what you're using in the assignment for the digit recognizer You're using 12 MFCCs plus one other feature The other feature we use is just the energy of the frame And it's very similar to the zeroth cepstral coefficient. It's just computed from the raw frame energy So you've actually got 13 features there energy plus 12 MFCCs But when you go and look in the configurations for the assignment and you go and look at your train models You'll find that you don't have 13 You've actually got 39 and it's part of the assignment to figure out how we got from 13 to 39 dimensional feature vectors And now it's obvious why we don't want to model covariance in the 39 dimensional feature space. We'd have 39 by 39 Dimensional covariance matrices. We'd have to estimate all of those covariances from data. So coming next we've got the Gaussian That's going to do all of the work. We've seen that the Gaussian can be seen as a generative model But a single Gaussian can just generate a single data point at a time But speech is not that. Speech is a sequence of observations, a sequence of feature vectors So we need a multivariate Gaussian that can generate one observation, one feature vector for one frame of speech And we need to put those Gaussians into a model that generates a sequence of observations And that's going to be the Hidden Markov Model and that's coming up next. That's the model for speech recognition in this course We've then got to think well, what are we going to model with each Hidden Markov Model?