Wechat: yu389741| Email: gisdqy@163.com

Shop:https://www.giserdqy.com/shop

通过squid轮循代理IP来采集网站


爬虫使用过程中经常会遇到反爬限制的网站,爬的快一点就会被封IP,那有没有一种简单的办法突破IP速度限制呢?

当然有一点爬虫经验的朋友肯定会想到使用代理突破防御.网络上有很多免费代理随便扫描一批就可以使用.每增加一个代理IP爬取速度就会增加一倍,所以使用代理爬网站是最经济实用的.

虽然pyspider支持代理配置,但是有很多代理那应该怎么配置呢?

其实作者已经想到了这个问题,并且在问答模块里也回复过多次网友的回复,使用squid就可以方便的实现代理ip的轮循.下面我们来看看squid怎么配置来轮循代理IP.

首先是先安装squid.ubuntu系统直接使用下面的命令安装squid:

sudo apt-get install squid3

如果是centos系统就使用下面的命令安装:

yum -y install squid

安装成功后修改配置文件为:下载配置文件

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny manager
http_access allow all
http_access allow localhost
http_access deny all
http_port 6666 #代理端口,可以自己指定
coredump_dir /var/spool/squid3
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .               0       20%     4320

#visible_hostname haha
via off
forwarded_for delete
#forwarded_for off
#follow_x_forwarded_for deny all
#request_header_access X-Forwarded-For deny all
#request_header_access From deny all
#request_header_access Referer deny all
#request_header_access User-Agent deny all

###cache_peer start 以下是代理IP列表

cache_peer 119.122.93.112 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=119.122.93.112820
cache_peer 219.128.136.242 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=219.128.136.242712
cache_peer 125.33.188.32 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=125.33.188.32867
cache_peer 58.100.106.139 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=58.100.106.139828
cache_peer 121.207.6.126 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=121.207.6.126574
cache_peer 14.209.24.31 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=14.209.24.31685
cache_peer 118.144.176.6 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=118.144.176.6621
cache_peer 104.236.31.8 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=104.236.31.8937
cache_peer 36.82.133.42 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=36.82.133.42776
cache_peer 110.246.104.84 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=110.246.104.84296
cache_peer 107.151.142.126 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.142.126894
cache_peer 50.81.91.68 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=50.81.91.68944
cache_peer 118.69.66.63 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=118.69.66.63818
cache_peer 177.130.59.66 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=177.130.59.66807
cache_peer 123.7.82.20 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.7.82.2025
cache_peer 59.104.194.23 parent 8088 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=59.104.194.23238
cache_peer 114.27.132.112 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=114.27.132.11270
cache_peer 122.226.141.67 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=122.226.141.67352
cache_peer 120.52.72.23 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.52.72.23762
cache_peer 58.242.248.6 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=58.242.248.6635
cache_peer 117.41.144.21 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=117.41.144.21791
cache_peer 218.10.97.115 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=218.10.97.115503
cache_peer 123.121.108.86 parent 9999 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.121.108.86916
cache_peer 117.168.73.233 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=117.168.73.233367
cache_peer 180.183.159.82 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=180.183.159.82713
cache_peer 162.243.133.13 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=162.243.133.13678
cache_peer 115.221.233.172 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=115.221.233.172856
cache_peer 112.92.206.113 parent 9999 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=112.92.206.113380
cache_peer 104.236.54.196 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=104.236.54.196892
cache_peer 121.238.231.97 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=121.238.231.9752
cache_peer 121.139.156.58 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=121.139.156.58256
cache_peer 124.202.180.6 parent 8118 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.202.180.6757
cache_peer 106.58.57.116 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=106.58.57.116650
cache_peer 119.18.234.60 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=119.18.234.6072
cache_peer 192.232.243.59 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=192.232.243.59959
cache_peer 61.166.56.177 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.166.56.177170
cache_peer 60.11.41.69 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.11.41.69848
cache_peer 107.151.142.117 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.142.117371
cache_peer 203.88.172.70 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=203.88.172.70197
cache_peer 60.191.158.211 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.191.158.211928
cache_peer 179.177.223.114 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=179.177.223.114454
cache_peer 171.5.136.68 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=171.5.136.68656
cache_peer 118.69.168.101 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=118.69.168.10157
cache_peer 1.9.166.132 parent 800 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=1.9.166.132855
cache_peer 218.7.170.190 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=218.7.170.190295
cache_peer 124.193.51.249 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.193.51.249165
cache_peer 223.199.45.13 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=223.199.45.13135
cache_peer 138.19.5.196 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=138.19.5.196957
cache_peer 1.82.216.134 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=1.82.216.134996
cache_peer 118.144.154.253 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=118.144.154.25328
cache_peer 104.236.124.96 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=104.236.124.96253
cache_peer 120.52.72.22 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.52.72.2261
cache_peer 128.199.132.138 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=128.199.132.138986
cache_peer 139.196.233.180 parent 21193 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=139.196.233.180406
cache_peer 125.113.132.132 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=125.113.132.1326
cache_peer 59.108.201.239 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=59.108.201.239180
cache_peer 111.206.81.248 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=111.206.81.248794
cache_peer 101.201.235.141 parent 8000 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=101.201.235.141452
cache_peer 58.242.248.5 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=58.242.248.5811
cache_peer 110.231.144.72 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=110.231.144.72172
cache_peer 107.151.152.212 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.152.212335
cache_peer 179.242.95.178 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=179.242.95.178439
cache_peer 180.175.240.83 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=180.175.240.83748
cache_peer 202.171.253.72 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=202.171.253.7293
cache_peer 113.106.213.162 parent 9797 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=113.106.213.162513
cache_peer 60.191.163.147 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.191.163.147256
cache_peer 182.42.216.120 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=182.42.216.120329
cache_peer 219.155.96.170 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=219.155.96.170321
cache_peer 118.244.239.2 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=118.244.239.2456
cache_peer 223.14.217.189 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=223.14.217.189243
cache_peer 123.57.190.51 parent 7777 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.57.190.51441
cache_peer 54.169.237.120 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=54.169.237.120300
cache_peer 58.20.249.157 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=58.20.249.157390
cache_peer 107.151.152.222 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.152.222473
cache_peer 123.161.70.5 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.161.70.512
cache_peer 111.252.221.24 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=111.252.221.24286
cache_peer 104.233.88.225 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=104.233.88.225846
cache_peer 178.49.228.101 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=178.49.228.101601
cache_peer 58.19.222.139 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=58.19.222.139438
cache_peer 60.191.147.202 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.191.147.20257
cache_peer 45.58.54.233 parent 25 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=45.58.54.233900
cache_peer 124.193.58.10 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.193.58.1080
cache_peer 182.48.113.11 parent 8088 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=182.48.113.1120
cache_peer 210.91.48.121 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=210.91.48.121353
cache_peer 61.53.65.54 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.53.65.54116
cache_peer 66.186.2.163 parent 443 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=66.186.2.163185
cache_peer 93.174.55.82 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=93.174.55.82398
cache_peer 115.226.215.24 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=115.226.215.24641
cache_peer 5.196.61.125 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=5.196.61.125734
cache_peer 218.103.42.105 parent 1080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=218.103.42.10528
cache_peer 183.19.46.57 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=183.19.46.5744
cache_peer 182.116.253.167 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=182.116.253.167414
cache_peer 111.67.202.83 parent 808 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=111.67.202.83193
cache_peer 220.248.229.45 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=220.248.229.45677
cache_peer 61.223.127.152 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.223.127.152617
cache_peer 112.112.70.115 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=112.112.70.115412
cache_peer 23.94.244.247 parent 25 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=23.94.244.24718
cache_peer 182.253.201.77 parent 10000 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=182.253.201.77678
cache_peer 46.101.81.85 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=46.101.81.85330
cache_peer 61.19.250.51 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.19.250.51113
cache_peer 121.228.20.236 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=121.228.20.236444
cache_peer 5.189.142.160 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=5.189.142.160801
cache_peer 41.33.22.186 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=41.33.22.186764
cache_peer 219.131.111.235 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=219.131.111.235674
cache_peer 113.16.186.245 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=113.16.186.245192
cache_peer 60.194.100.51 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.194.100.51572
cache_peer 113.107.57.76 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=113.107.57.7655
cache_peer 121.14.36.38 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=121.14.36.38320
cache_peer 124.133.240.88 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.133.240.88846
cache_peer 1.255.53.81 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=1.255.53.81913
cache_peer 123.139.59.85 parent 9999 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.139.59.8563
cache_peer 101.30.208.216 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=101.30.208.216105
cache_peer 123.56.45.102 parent 82 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.56.45.102974
cache_peer 119.180.92.124 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=119.180.92.124917
cache_peer 84.119.63.243 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=84.119.63.243707
cache_peer 202.103.215.199 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=202.103.215.199797
cache_peer 200.29.191.149 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=200.29.191.149412
cache_peer 178.33.11.87 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=178.33.11.87157
cache_peer 107.151.136.222 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.136.222351
cache_peer 124.120.30.10 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.120.30.10511
cache_peer 116.25.80.164 parent 8118 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=116.25.80.164416
cache_peer 36.7.172.18 parent 82 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=36.7.172.18561
cache_peer 107.151.152.211 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.152.21171
cache_peer 60.191.159.86 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.191.159.86462
cache_peer 107.151.136.219 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.136.21963
cache_peer 124.16.70.11 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.16.70.1185
cache_peer 202.77.57.124 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=202.77.57.12412
cache_peer 178.18.25.184 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=178.18.25.184670
cache_peer 123.30.238.16 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.30.238.16646
cache_peer 120.28.45.202 parent 8090 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.28.45.202230
cache_peer 1.82.216.135 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=1.82.216.135556
cache_peer 113.106.213.162 parent 9797 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=113.106.213.162531
cache_peer 103.54.147.132 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=103.54.147.132130
cache_peer 36.72.22.93 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=36.72.22.93296
cache_peer 107.151.142.122 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.142.122667
cache_peer 60.191.164.83 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=60.191.164.8330
cache_peer 183.19.48.88 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=183.19.48.88741
cache_peer 123.120.21.229 parent 9999 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=123.120.21.229516
cache_peer 188.166.194.193 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=188.166.194.193340
cache_peer 177.17.231.81 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=177.17.231.81204
cache_peer 82.196.10.29 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=82.196.10.29451
cache_peer 221.226.67.202 parent 8118 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=221.226.67.202404
cache_peer 120.132.71.212 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.132.71.212125
cache_peer 106.38.251.62 parent 8088 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=106.38.251.62671
cache_peer 183.88.73.178 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=183.88.73.178311
cache_peer 106.75.128.89 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=106.75.128.89490
cache_peer 61.75.2.124 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.75.2.124394
cache_peer 120.52.72.55 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.52.72.55753
cache_peer 27.131.47.132 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=27.131.47.132988
cache_peer 124.192.106.247 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.192.106.247426
cache_peer 110.179.89.175 parent 81 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=110.179.89.175194
cache_peer 113.107.57.76 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=113.107.57.76804
cache_peer 180.107.84.131 parent 8888 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=180.107.84.131113
cache_peer 46.101.23.211 parent 3129 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=46.101.23.211958
cache_peer 202.100.167.142 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=202.100.167.142917
cache_peer 124.126.118.223 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=124.126.118.223828
cache_peer 118.253.82.126 parent 9797 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=118.253.82.1263
cache_peer 110.7.106.143 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=110.7.106.143872
cache_peer 180.250.32.66 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=180.250.32.66706
cache_peer 120.27.210.131 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.27.210.131656
cache_peer 183.7.88.110 parent 8118 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=183.7.88.110224
cache_peer 139.196.73.101 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=139.196.73.101952
cache_peer 61.160.212.74 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.160.212.74367
cache_peer 120.236.148.199 parent 2226 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=120.236.148.19917
cache_peer 61.166.56.177 parent 3128 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=61.166.56.17748
cache_peer 171.116.209.27 parent 9797 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=171.116.209.27409
cache_peer 182.253.123.32 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=182.253.123.32619
cache_peer 107.151.136.202 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.136.20278
cache_peer 114.130.13.60 parent 8080 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=114.130.13.60764
cache_peer 101.231.250.102 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=101.231.250.102390
cache_peer 107.151.136.194 parent 80 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=107.151.136.194798
cache_peer 139.196.234.11 parent 21193 0 no-query weighted-round-robin weight=1 connect-fail-limit=2 allow-miss max-conn=5 name=139.196.234.11270

never_direct allow all

配置文件一般是在/etc/squid3/下的squid.conf文件,请对应修改.ubuntu系统可以直接全部替换,centos对应修改就可以了.重启后就可以生效了.可以把浏览器的HTTP代理设置成127.0.0.1:6666或是服务器的IP就可以,设置好后就可以访问能显示IP的网站查看了.如:http://httpbin.org/get

如果代理IP是免费采集的就需要写个脚本自动采集验证自动生成配置文件,然后重新载入配置文件或重启squid.

pyspider使用的时候只需要设置代理服务器为你配置好的服务器的IP就可以了,如下面项目代码.是不是很简单呢?

class Handler(BaseHandler):
    crawl_config = {
        'proxy': '127.0.0.1:6666'
    }

感谢@开水泡脚丫提供配置文件


转载自:http://www.pyspider.cn/jiaocheng/squid%EF%BC%BFproxy-13.html