notebook

小车杆滑行#

问题：如果彼得想要逃离狼的追捕，他需要比狼移动得更快。我们将探讨彼得如何学习滑行，特别是如何通过Q学习保持平衡。

首先，让我们安装gym并导入所需的库：

In [1]:
import sys
!pip install gym pygame

import gym
import matplotlib.pyplot as plt
import numpy as np
import random
Defaulting to user installation because normal site-packages is not writeable
Collecting gym
  Downloading gym-0.25.0.tar.gz (720 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 720.4/720.4 KB 3.3 MB/s eta 0:00:0000:0100:01
[?25h  Installing build dependencies ... [?25ldone
[?25h  Getting requirements to build wheel ... [?25ldone
[?25h  Preparing metadata (pyproject.toml) ... [?25ldone
[?25hRequirement already satisfied: numpy>=1.18.0 in /usr/lib/python3/dist-packages (from gym) (1.21.5)
Collecting gym-notices>=0.0.4
  Downloading gym_notices-0.0.7-py3-none-any.whl (2.7 kB)
Collecting cloudpickle>=1.2.0
  Downloading cloudpickle-2.1.0-py3-none-any.whl (25 kB)
Building wheels for collected packages: gym
  Building wheel for gym (pyproject.toml) ... [?25ldone
[?25h  Created wheel for gym: filename=gym-0.25.0-py3-none-any.whl size=824430 sha256=3f4ed647f1d12814bb457f7d83a7ccd0f682d12a0259ca07b7fab0db5100fc6e
  Stored in directory: /home/leo/.cache/pip/wheels/c0/3c/33/32d86254a5bd554f5f07759ae1794646e490dd5fa81ebdcda3
Successfully built gym
Installing collected packages: gym-notices, cloudpickle, gym
Successfully installed cloudpickle-2.1.0 gym-0.25.0 gym-notices-0.0.7

创建一个平衡杆环境#

In [2]:
env = gym.make("CartPole-v1")
print(env.action_space)
print(env.observation_space)
print(env.action_space.sample())
Discrete(2)
Box([-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38], [4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38], (4,), float32)
1

/home/leo/.local/lib/python3.10/site-packages/gym/core.py:329: DeprecationWarning: WARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(
/home/leo/.local/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: WARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(

要了解环境如何运行，让我们进行一个100步的短模拟。

In [3]:
env.reset()

for i in range(100):
   env.render()
   env.step(env.action_space.sample())
env.close()
/home/leo/.local/lib/python3.10/site-packages/gym/core.py:57: DeprecationWarning: WARN: You are calling render method, but you didn't specified the argument render_mode at environment initialization. To maintain backward compatibility, the environment will render in human mode.
If you want to render in human mode, initialize the environment in this way: gym.make('EnvName', render_mode='human') and don't call the render method.
See here for more information: https://www.gymlibrary.ml/content/api/
  deprecation(

DependencyNotInstalled: pygame is not installed, run `pip install gym[classic_control]`
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py in _render(self, mode)
    221         try:
--> 222             import pygame
    223             from pygame import gfxdraw

ModuleNotFoundError: No module named 'pygame'

During handling of the above exception, another exception occurred:

DependencyNotInstalled                    Traceback (most recent call last)
/tmp/ipykernel_32716/4123126963.py in <module>
      2 
      3 for i in range(100):
----> 4    env.render()
      5    env.step(env.action_space.sample())
      6 env.close()

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
     64                 )
     65 
---> 66             return render_func(self, *args, **kwargs)
     67 
     68         return render

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
    429     def render(self, *args, **kwargs):
    430         """Renders the environment."""
--> 431         return self.env.render(*args, **kwargs)
    432 
    433     def close(self):

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
     64                 )
     65 
---> 66             return render_func(self, *args, **kwargs)
     67 
     68         return render

~/.local/lib/python3.10/site-packages/gym/wrappers/order_enforcing.py in render(self, *args, **kwargs)
     49                 "set `disable_render_order_enforcing=True` on the OrderEnforcer wrapper."
     50             )
---> 51         return self.env.render(*args, **kwargs)
     52 
     53     @property

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
     64                 )
     65 
---> 66             return render_func(self, *args, **kwargs)
     67 
     68         return render

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
    429     def render(self, *args, **kwargs):
    430         """Renders the environment."""
--> 431         return self.env.render(*args, **kwargs)
    432 
    433     def close(self):

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
     64                 )
     65 
---> 66             return render_func(self, *args, **kwargs)
     67 
     68         return render

~/.local/lib/python3.10/site-packages/gym/wrappers/env_checker.py in render(self, *args, **kwargs)
     51         if self.checked_render is False:
     52             self.checked_render = True
---> 53             return env_render_passive_checker(self.env, *args, **kwargs)
     54         else:
     55             return self.env.render(*args, **kwargs)

~/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py in env_render_passive_checker(env, *args, **kwargs)
    322             )
    323 
--> 324     result = env.render(*args, **kwargs)
    325 
    326     # TODO: Check that the result is correct

~/.local/lib/python3.10/site-packages/gym/core.py in render(self, *args, **kwargs)
     64                 )
     65 
---> 66             return render_func(self, *args, **kwargs)
     67 
     68         return render

~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py in render(self, mode)
    215             return self.renderer.get_renders()
    216         else:
--> 217             return self._render(mode)
    218 
    219     def _render(self, mode="human"):

~/.local/lib/python3.10/site-packages/gym/envs/classic_control/cartpole.py in _render(self, mode)
    223             from pygame import gfxdraw
    224         except ImportError:
--> 225             raise DependencyNotInstalled(
    226                 "pygame is not installed, run `pip install gym[classic_control]`"
    227             )

DependencyNotInstalled: pygame is not installed, run `pip install gym[classic_control]`

在模拟过程中，我们需要获取观测值以决定如何行动。实际上，step函数会返回当前的观测值、奖励函数，以及done标志，该标志指示是否有必要继续模拟：

In [4]:
env.reset()

done = False
while not done:
   env.render()
   obs, rew, done, info = env.step(env.action_space.sample())
   print(f"{obs} -> {rew}")
env.close()
[ 0.03044442 -0.19543914 -0.04496216  0.28125618] -> 1.0
[ 0.02653564 -0.38989186 -0.03933704  0.55942606] -> 1.0
[ 0.0187378  -0.19424049 -0.02814852  0.25461393] -> 1.0
[ 0.01485299 -0.38894946 -0.02305624  0.53828712] -> 1.0
[ 0.007074   -0.19351108 -0.0122905   0.23842953] -> 1.0
[ 0.00320378  0.00178427 -0.00752191 -0.05810469] -> 1.0
[ 0.00323946  0.19701326 -0.008684   -0.35315131] -> 1.0
[ 0.00717973  0.00201587 -0.01574703 -0.06321931] -> 1.0
[ 0.00722005  0.19736001 -0.01701141 -0.36082863] -> 1.0
[ 0.01116725  0.39271958 -0.02422798 -0.65882671] -> 1.0
[ 0.01902164  0.19794307 -0.03740452 -0.37387001] -> 1.0
[ 0.0229805   0.39357584 -0.04488192 -0.67810827] -> 1.0
[ 0.03085202  0.58929164 -0.05844408 -0.98457719] -> 1.0
[ 0.04263785  0.78514572 -0.07813563 -1.2950295 ] -> 1.0
[ 0.05834076  0.98116859 -0.10403622 -1.61111521] -> 1.0
[ 0.07796413  0.78741784 -0.13625852 -1.35259196] -> 1.0
[ 0.09371249  0.98396202 -0.16331036 -1.68461179] -> 1.0
[ 0.11339173  0.79106371 -0.1970026  -1.44691436] -> 1.0
[ 0.12921301  0.59883361 -0.22594088 -1.22169133] -> 1.0

我们可以获取这些数字的最小值和最大值：

In [5]:
print(env.observation_space.low)
print(env.observation_space.high)
[-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38]
[4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38]
In [6]:
def discretize(x):
    return tuple((x/np.array([0.25, 0.25, 0.01, 0.1])).astype(np.int))

让我们也探索使用分箱的其他离散化方法：

In [7]:
def create_bins(i,num):
    return np.arange(num+1)*(i[1]-i[0])/num+i[0]

print("Sample bins for interval (-5,5) with 10 bins\n",create_bins((-5,5),10))

ints = [(-5,5),(-2,2),(-0.5,0.5),(-2,2)] # intervals of values for each parameter
nbins = [20,20,10,10] # number of bins for each parameter
bins = [create_bins(ints[i],nbins[i]) for i in range(4)]

def discretize_bins(x):
    return tuple(np.digitize(x[i],bins[i]) for i in range(4))
Sample bins for interval (-5,5) with 10 bins
 [-5. -4. -3. -2. -1.  0.  1.  2.  3.  4.  5.]

现在让我们运行一个简短的模拟，并观察那些离散的环境值。

In [8]:
env.reset()

done = False
while not done:
   #env.render()
   obs, rew, done, info = env.step(env.action_space.sample())
   #print(discretize_bins(obs))
   print(discretize(obs))
env.close()
(0, 0, -1, -3)
(0, 0, -2, 0)
(0, 0, -2, -3)
(0, 1, -3, -6)
(0, 2, -4, -9)
(0, 3, -6, -12)
(0, 2, -8, -9)
(0, 3, -10, -13)
(0, 4, -13, -16)
(0, 4, -16, -19)
(0, 4, -20, -17)
(0, 4, -24, -20)

Q-表结构#

In [9]:
Q = {}
actions = (0,1)

def qvalues(state):
    return [Q.get((state,a),0) for a in actions]
In [10]:
# hyperparameters
alpha = 0.3
gamma = 0.9
epsilon = 0.90
In [11]:
def probs(v,eps=1e-4):
    v = v-v.min()+eps
    v = v/v.sum()
    return v

Qmax = 0
cum_rewards = []
rewards = []
for epoch in range(100000):
    obs = env.reset()
    done = False
    cum_reward=0
    # == do the simulation ==
    while not done:
        s = discretize(obs)
        if random.random()<epsilon:
            # exploitation - chose the action according to Q-Table probabilities
            v = probs(np.array(qvalues(s)))
            a = random.choices(actions,weights=v)[0]
        else:
            # exploration - randomly chose the action
            a = np.random.randint(env.action_space.n)

        obs, rew, done, info = env.step(a)
        cum_reward+=rew
        ns = discretize(obs)
        Q[(s,a)] = (1 - alpha) * Q.get((s,a),0) + alpha * (rew + gamma * max(qvalues(ns)))
    cum_rewards.append(cum_reward)
    rewards.append(cum_reward)
    # == Periodically print results and calculate average reward ==
    if epoch%5000==0:
        print(f"{epoch}: {np.average(cum_rewards)}, alpha={alpha}, epsilon={epsilon}")
        if np.average(cum_rewards) > Qmax:
            Qmax = np.average(cum_rewards)
            Qbest = Q
        cum_rewards=[]
0: 108.0, alpha=0.3, epsilon=0.9
In [20]:
plt.plot(rewards)
[<matplotlib.lines.Line2D at 0x2814bb79788>]
Notebook 输出图像

从这个图表中无法得出任何结论，因为由于随机训练过程的性质，训练会话的长度差异很大。为了更好地理解这个图表，我们可以对一系列实验计算移动平均值，比如说100。这可以通过使用np.convolve方便地完成：

In [22]:
def running_average(x,window):
    return np.convolve(x,np.ones(window)/window,mode='valid')

plt.plot(running_average(rewards,100))
[<matplotlib.lines.Line2D at 0x2814cc63488>]
Notebook 输出图像

调整超参数并观察结果#

现在，实际观察训练好的模型表现会很有趣。让我们运行模拟，并按照训练时相同的动作选择策略进行操作：根据 Q-Table 中的概率分布进行采样：

In [23]:
obs = env.reset()
done = False
while not done:
   s = discretize(obs)
   env.render()
   v = probs(np.array(qvalues(s)))
   a = random.choices(actions,weights=v)[0]
   obs,_,done,_ = env.step(a)
env.close()

将结果保存为动画 GIF#

如果你想给朋友留下深刻印象，可以考虑发送平衡杆的动画 GIF 图片。为此，我们可以调用 env.render 来生成图像帧，然后使用 PIL 库将这些帧保存为动画 GIF：

In [26]:
from PIL import Image
obs = env.reset()
done = False
i=0
ims = []
while not done:
   s = discretize(obs)
   img=env.render(mode='rgb_array')
   ims.append(Image.fromarray(img))
   v = probs(np.array([Qbest.get((s,a),0) for a in actions]))
   a = random.choices(actions,weights=v)[0]
   obs,_,done,_ = env.step(a)
   i+=1
env.close()
ims[0].save('images/cartpole-balance.gif',save_all=True,append_images=ims[1::2],loop=0,duration=5)
print(i)
360

免责声明：
本文档使用AI翻译服务Co-op Translator进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。应以原始语言的文档作为权威来源。对于重要信息，建议使用专业人工翻译。我们不对因使用此翻译而产生的任何误解或误读承担责任。