1、

设定浏览器列表。

首先需要定义各种不同类型的浏览器，统一保存于一个列表中。我们将其定义在settings.py中

2、

MY_USER_AGENT = [

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",\

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",\

“Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,\.......]

3、

将settings.py中设置固定浏览器代码删除或注释掉。

# Crawl responsibly by identifying yourself (and your website) on the user-agent

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0;Win64; x64) " \

"AppleWebKit/537.36 (KHTML, like Gecko) " \

"Chrome/68.0.3440.106 Safari/537.36"