5 – Building The Network

Hello everyone. Welcome back. So now we’re going to actually start building the network. So here we can see the general structure of the network that we’re going to build. So we have our inputs, and as I said before this, we’re not actually going to be doing this you know one hot encoding, we’re going to be passing in integers, but so we’re going to pass them to this hidden layer which is our embedding layer. And this is just linear neurons so we’re not, you know putting the activation on them at all. They’re just linear. And then, this is going to an output softmax layer. So again remember that we’re, we’re passing in like some current linear text and then, and then target words, and so we’re going to be trying to predict our target words using this softmax layer. Then when we train everything up, what’s going to happen is that our, our hidden layer is going to form these you know vector representations of the words that contain some semantic meaning. And that’s, that’s what we’re really interested in. We’re only, we only really care about these these representations, that’s what we’re doing, it’s like we’re, we’re changing these words into vectors. So what that means is that when we’re done training we can just get rid of this softmax layer, that we don’t really like care about it anymore. It’s just there to, to train up the, basically like the weight between the inputs and the hidden layer. Okay, so the first thing you’re going to do here is build the inputs. So here, you’re going to create the inputs using tensor flow placeholders. So we’re going to be passing in integers and so then you need to set the data type to tf.int32. And then the batches that we’re passing in are going to have random sizes so you’ll need to leave the batch size arbitrary, so you can just set that to none. And then tensor flow will figure it out when you pass in the batches. And then to make things work later on with some of the code, the labels, the targets, this placeholder needs to have a second dimension that you either set to none or one. Okay, so that is the input nodes. So down here we’re going to look at our embedding layer. So again here you’re going to create your embedding layer. And so the first thing you do is create the embedding matrix. So this is going to have, you know the size of the number of words in your vocabulary, by your desired embedding dimensions. So for example if you have 10000 words and you have 300 hidden units which is your embedding dimension then the matrix will be 10000 by 300. So then, here you’re going to be setting the number of embedding features and the embedding dimension and then create the embedding weight matrix here. So remember that you do this with tf.variable. Okay, now to do the actual look up, we use tf.nn.embedding_lookup. So you can read the documentation here but basically you can pass in the the weight matrix and then you pass in some integer and then it does the look up, or you can pass in a tensor integers and it’s going to do the look up on all those tensors and then you get your hidden layer values. So I’ll let you work on this. And so here you’ll be doing like creating the inputs and creating the embedding layer. So if you need help with this check out my solution, and I’ll have a video with my solution next.

%d 블로거가 이것을 좋아합니다: