Requêtes Wikipedia + Grequests

Question

J'essaie d'interroger ~ 50 pages Wikipedia. J'ai utilisé le paquet de requêtes pour faire des requêtes GET, mais j'ai travaillé sur l'implémentation de grequests car j'entends dire qu'il a de meilleures performances.

L'amélioration de la performance est vraiment minime pour moi. Est-ce que je fais quelque chose de mal?

import requests 
import grequests 
from urllib.parse import quote 
from time import time 

url = 'https://en.wikipedia.org/w/api.php?action=query&titles={0}&prop=pageprops&ppprop=disambiguation&format=json' 
titles = ['Harriet Tubman', 'Car', 'Underground Railroad', 'American Civil War', 'Kate Larson'] 
urls = [url.format(quote(title)) for title in titles] 

def sync_test(urls): 
    results = [] 
    s = time() 
    for url in urls: 
     results.append(requests.get(url)) 
    e = time() 
    return e-s 

def async_test(urls): 
    s = time() 
    results = grequests.map((grequests.get(url) for url in urls)) 
    e = time() 
    return e-s 

def iterate(urls, num): 
    sync_time = 0 
    async_time = 0 
    for i in range(num): 
     sync_time += sync_test(urls) 
     async_time += async_test(urls) 
    print("sync_time: {}\nasync_time: {}".format(sync_time, async_time)) 

sortie: sync_time: 8,945282936096191 async_time: 7,97578239440918

Merci!

Answer 1

import requests 
import grequests 
from urllib.parse import quote 
from time import time 

url = 'https://en.wikipedia.org/w/api.php?action=query&titles={0}&prop=pageprops&ppprop=disambiguation&format=json' 
titles = ['Harriet Tubman', 'Car', 'Underground Railroad', 'American Civil War', 'Kate Larson'] 
urls = [url.format(title) for title in titles] 

def sync_test(urls): 
    results = [] 
    s = time() 
    for url in urls: 
     results.append(requests.get(url)) 
    e = time() 
    return e-s 

def async_test(urls): 
    s = time() 
    results = grequests.map((grequests.get(url) for url in urls)) 
    e = time() 
    return e-s 

def iterate(urls, num): 
    sync_time = 0 
    async_time = 0 
    for i in range(num): 
     sync_time += sync_test(urls) 
     async_time += async_test(urls) 
    print("sync_time: {}\nasync_time: {}".format(sync_time, async_time)) 

if __name__ == '__main__': 
    iterate(urls,10) 

Cela me donne:

sync_time: 22.14458918571472 
async_time: 4.846134662628174 

Process finished with exit code 0 

Je ne vois pas de problème ici