多线程 + requests 并发请求
44.10 多线程 + requests 并发请求
多个 URL 同时请求时,I/O 等待期间线程可切换,总耗时接近最慢单个而非相加。
需 pip install requests;注意控制并发数,勿对同一站点造成过大压力。
更规范做法:ThreadPoolExecutor + submit(fetch, url)。
并发 GET 请求
# ========================================
# 示例:多线程 requests
# 需 pip install requests
# ========================================
import threading
import requests
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/1',
'https://httpbin.org/get'
]
def fetch(url):
r = requests.get(url, timeout=15)
print(url[:40], '...', r.status_code)
threads = [threading.Thread(target=fetch, args=(u,)) for u in urls]
for t in threads:
t.start()
for t in threads:
t.join()
print('全部完成')