py3-5-aiohttp-百万请求？

2017-01-03 09:52

说明

  • 一直想自己写个基于python的异步压测客户端，无意发现py3的aiohttp，找到了一篇英文文章说的就是构造百万级别http请求
  • http://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html
  • 作者对各个请求给服务器造成的情况作出了详细的对比。我就偷个懒没有。。。
  • 基于py3.5 3.4的语法请自行度娘

例子一

import asyncio
from aiohttp import ClientSession
# 　你使用async以及await关键字将函数异步化
async def fetch(url):
    async with ClientSession() as session:
        async with session.get(url) as response:
            return await response.read()
async def run(loop,  r):
    url = "http://gc.ditu.aliyun.com/geocoding?a=苏州市"
    tasks = []
    for i in range(r):
        task = asyncio.ensure_future(fetch(url.format(i)))
        tasks.append(task)

    responses = await asyncio.gather(*tasks)
    # 注意asyncio.gather()的用法，它搜集所有的Future对象，然后等待他们返回。
    # print(json.loads(responses[0].decode()))
    print(len(responses))

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(loop, 800))
loop.run_until_complete(future)
  • future = asyncio.ensure_future(run(loop, 800)) 这里 构造1000个请求时，就报too many open files loop is not close 作者也遇到此问题，说是本机的socket 端口用光了？很是怀疑

例子二 asyncio.Semaphore解决报错问题

import asyncio
from aiohttp import ClientSession

async def fetch(url):
    async with ClientSession() as session:
        async with session.get(url) as response:
            return await response.read()


async def bound_fetch(sem, url):
    async with sem:
        await fetch(url)

async def run(loop,  r):
    url = "http://gc.ditu.aliyun.com/geocoding?a=苏州市"
    tasks = []
    # create instance of Semaphore
    sem = asyncio.Semaphore(100)
    for i in range(r):
        # pass Semaphore to every GET request
        task = asyncio.ensure_future(bound_fetch(sem, url.format(i)))
        tasks.append(task)

    responses = await asyncio.gather(*tasks)
    print(responses)
number = 100000
loop = asyncio.get_event_loop()

future = asyncio.ensure_future(run(loop, number))
loop.run_until_complete(future)
  • 然而我用asyncio.Semaphore时，发现请求不成功，已经发了邮件给作者，没有回我。