3-5. Actor-Critic Methods

Unlike in DQN, A3C does not use a replay buffer. The main reason we needed a replay buffer was so that we could decorrelate experienced topple. Let me explain. In reinforcement learning, an agent collects experience in a sequential manner. The experience collected at time step t plus 1 will be correlated to the experience … Read more

A3C stands for Asynchronous Advantage Actor-Critic. As you can probably infer from the name, we’ll be calculating the advantage function. A Pi essay, and the critic will be learning to estimate V Pi to help with that just as before. If you’re using images as inputs to your agent, A3C can use a single convolutional … Read more

You know now that an actor-critic agent is an agent that uses function approximation to learn a policy any value function. So, we will then use two neural networks; one for the actor and one for the critic. The critic will learn to evaluate the state value function V Pi using the TD estimate. Using … Read more

Now that you have some foundational concepts down, let me give you some intuition. Let’s say you want to get better at tennis. The actor or policy-based approaching you roughly learns this way. You play a bunch of matches. You then go home, lay on the couch, and commit to yourself to do more of … Read more

You now know that the Monte-Carlo estimate is unbiased but has high variance, and that the TD estimate has low variance but it is biased. What are these facts good for? See when you study ring force, you learned that the return G was calculated as the total discounter return. This way of calculating G, … Read more

Let’s explore two very distinct and complimentary ways for estimating expected returns. On the one hand, you’d have the Monte-Carlo estimate. The Monte-Carlo estimate consists of rolling out an episode in calculating the discounter total reward from the rewards sequence. For example, in an episode A, you start in state S_t, take action A_t. The … Read more

Let’s talk about bias and variance. In machine learning, we’re often presented there with a trade off between bias and variance. Let me give you some intuition first. Let’s say you’re a practicing your soccer shooting skills. The thing you want to do is to put the ball in the top right corner of the … Read more

Actor-critic methods are at the intersection of value-based methods such as DQN and policy-based methods such as reinforce. If a deep reinforcement learning agent uses a deep neural network to approximate a value function, the agent is said to be value-based. If an agent uses a deep neural network to approximate a policy, the agent … Read more

Well, this is the end of the actor-critic methods lesson. That was a lot, I know. But you’ll soon have a chance to put everything into practice, and that should help you cement concepts. In this lesson, you learned about actor-critic methods, which are simply a way to reduce the variance in policy-based methods. You … Read more

Hey guys, we got here. So, I wanted to show you now the DDPG. Now, remember DDPG’s somewhat, it’s questionable if is say an actor-critic or not. But nevertheless it’s a very important algorithm and the guys that created the algorithm said this is actor-critics. So, we’re going to just go with that. All right. … Read more

Two other interesting aspects of DDPG are first, the use of a replay buffer, and second, the soft updates to the target networks. You already know how the replay buffer part works. I just wanted to mention that DDPG uses a replay buffer. But the soft updates are a bit different. In DQN, you have … Read more

DDPG is a different kind of actor-critic method. In fact, it could be seen as approximate DQN, instead of an actual actor critic. The reason for this is that the critic in DDPG, is used to approximate the maximizer over the Q values of the next state, and not as a learned baseline, as we … Read more

There is another way for estimating expected returns called the lambda return. The intuition goes this way. Say after you try and step bootstrapping you realize that numbers of n larger than one often perform better. But it’s still hard to tell what the number should be. Should it be a two, three, six or … Read more

Hey guys, you are here, I wanted to do a code walkthrough video. I wanted to show a very nice implementation of the A2C algorithm by Shangtong Zang. He’s a student in University of Alberta. He’s a student of [inaudible] Professor [inaudible]. He actually was the guy that ported all the code for the reinforcement … Read more

You may be wondering what the asynchronous part in A3C is about? Recall, Asynchronous Advantage Actor-Critic. Let me explain. A3C accumulates gradient updates and applies those updates asynchronously to a global neuronetwork. Each agent in simulation does this at its own time. So, the agents use a local copy of the network to collect experience, … Read more

In case you’re not clear on what on-policy versus off-policy learning is, let me explain that real quick. On-policy learning is when the policy use for interacting with the environment is also the policy being learned. Off-policy learning is when the policy used for interacting with the environment is different than the policy being learned. … Read more

Hi, my name is Miguel. I’m a software engineer for Lockheed Martin working on autonomous systems. Lockheed Martin is a global security in aerospace company engaged in the production of advanced technology systems. The majority of our business is naturally, with the US Department of Defense. At autonomous systems in Littleton, Colorado, we do all … Read more