informations manquantes dans les données d'exploration

Question

Je veux utiliser R pour analyser toutes les nouvelles (titre, URL et texte) liées à AlphaGo dans XXX, et l'URL de la page est http://www.xxxxxx.com/search/?q=AlphaGo. Voici mon code:

url <- "http://www.xxxxxx.com/search/?q=AlphaGo" 
info <- debugGatherer() 
handle <- getCurlHandle(cookiejar ="", 
         #turn the page 
         followlocation = TRUE, 
         autoreferer = TRUE, 
         debugfunc = info$update, 
         verbose = TRUE, 
         httpheader = list(
          from = "eddie@r-datacollection.com", 
          'user-agent' = str_c(R.version$version.string, 
               ",",R.version$platform) 
         )) 
html <- getURL(url,curl=handle,header = TRUE) 
parsedpage <- htmlParse(html) 

Cependant, lorsque j'utilise le code

xpathSApply(parsedpage,"//h3//a",xmlGetAttr,"href") 

pour vérifier si j'ai trouvé le code cible, je trouve que tout le contenu de l'information des nouvelles connexes manque. Ensuite, j'ai trouvé que le DOM elements (Chrome est ce que j'ai utilisé) après avoir appuyé sur F12 contient les informations que je veux, tandis que rien dans sources (ce qui est vraiment désordonné comme tous les éléments sont empilés ensemble). Donc, je change mon code à:

parsed_page <- htmlTreeParse(file = url,asTree = T) 

avec l'espoir d'acquérir le dom tree à la place. Encore, cette fois l'information est manquante, ce que je trouve est que toutes les informations manquantes sont les informations pliées dans le DOM elements (je n'ai jamais rencontré cette situation auparavant).

Une idée de comment le problème se produit et comment je pourrais résoudre ce problème?

Answer 1

Avec les pensées @Colin fourni, j'ai essayé de suivre le code d'origine. Donc, je code comme suit pour le contenu dynamique dans le fichier JSON avec le paquet RJSONIO

url <- "https://search.xxxxxx.io/content?q=AlphaGo" 
content <- fromJSON(url) 
content1 <- content$result 
content_result <- matrix(NA,10,5) 
for(i in 1:length(content1)){ 
    content_result[i,] <- c("CNN", content1[[i]]$firstPublishDate,ifelse(class(content1[[i]]$headline) != "NULL",content1[[i]]$headline,"NA"), 
         content1[[i]]$body,content1[[i]]$url) 
} 

Answer 2

Le problème ne provient pas de votre code. La page de résultats est générée dynamiquement, ainsi les liens et les textes ne sont pas disponibles en html simple dans la page de résultat (comme vous pouvez le voir si vous regardez le code source).

Il n'y a que 10 résultats, je vous suggère donc de créer manuellement une liste d'URL.

Je ne connais pas le paquet que vous avez utilisé dans ce code. Mais je vous suggère d'opter pour rvest, ce qui semble beaucoup plus simple que le paquet que vous avez utilisé.

Pour:

url <- "http://money.cnn.com/2017/05/25/technology/alphago-china-ai/index.html" 

library(rvest) 
library(tidyverse) 

url %>% 
    read_html() %>% 
    html_nodes(xpath = '//*[@id="storytext"]/p') %>% 
    html_text() 

[1] " A computer system that Google engineers trained to play the game Go beat the world's best human player Thursday in China. The victory was AlphaGo's second this week over Chinese professional Ke Jie, clinching the best-of-three series at the Future of Go Summit in Wuzhen. "         
[2] " Afterward, Google engineers said AlphaGo estimated that the first 50 moves -- by both players -- were virtually perfect. And the first 100 moves were the best anyone had ever played against AlphaGo's master version. "                       
[3] " Related: Google's man-versus-machine showdown is blocked in China "                                                             
[4] " \"What an amazing and complex game! Ke Jie pushed AlphaGo right to the limit,\" said DeepMind CEO Demis Hassabis on Twitter. DeepMind is a British artificial intelligence company that developed AlphaGo and was purchased by Google in 2014. "                  
[5] " DeepMind made a stir in January 2016 when it first announced it had used artificial intelligence to master Go, a 2,500-year-old game. Computer scientists had struggled for years to get computers to excel at the game. "                       
[6] " In Go, two players alternate placing white and black stones on a grid. The goal is to claim the most territory. To do so, you surround your opponent's pieces so that they're removed from the board. "                            
[7] " The board's 19-by-19 grid is so vast that it allows a near infinite combination of moves, making it tough for machines to comprehend. Games such as chess have come quicker to machines. "                               
[8] " Related: Elon Musk's new plan to save humanity from AI "                                                                
[9] " The Google engineers at DeepMind rely on deep learning, a trendy form of artificial intelligence that's driving remarkable gains in what computers are capable of. World-changing technologies that loom on the horizon, such as autonomous vehicles, rely on deep learning to effectively see and drive on roads. " 
[10] " AlphaGo's achievement is also a reminder of the steady improvement of machines' ability to complete tasks once reserved for humans. As machines get smarter, there are concerns about how society will be disrupted, and if all humans will be able to find work. "             
[11] " Historically, mankind's development of tools has always created new jobs that never existed before. But the gains in artificial intelligence are coming at a breakneck pace, which will likely accentuate upheaval in the short term. "                    
[12] " Related: Google uses AI to help diagnose breast cancer "                                                                
[13] " The 19-year-old Ke and AlphaGo will play a third match Saturday morning. The summit will also feature a match Friday in which five human players will team up against AlphaGo. "  

Meilleur

Colin