us.codecraft.webmagic.downloader

类 PhantomJSDownloader

java.lang.Object

us.codecraft.webmagic.downloader.AbstractDownloader

us.codecraft.webmagic.downloader.PhantomJSDownloader

所有已实现的接口:

Downloader

@ThreadSafe
public class PhantomJSDownloader
extends AbstractDownloader

this downloader is used to download pages which need to render the javascript

版本:

0.5.3

作者:

dolphineor@gmail.com

构造器概要

构造器

构造器和说明
PhantomJSDownloader()
PhantomJSDownloader(String phantomJsCommand) 添加新的构造函数，支持phantomjs自定义命令 example: phantomjs.exe 支持windows环境 phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误 /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException
PhantomJSDownloader(String phantomJsCommand, String crawlJsPath) 新增构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js crawl.js start -- var system = require('system'); var url = system.args[1]; var page = require('webpage').create(); page.settings.loadImages = false; page.settings.resourceTimeout = 5000; page.open(url, function (status) { if (status !

方法概要

限定符和类型	方法和说明
Page	download(Request request, Task task)
protected String	getPage(Request request)
int	getRetryNum()
PhantomJSDownloader	setRetryNum(int retryNum)
void	setThread(int threadNum)

从类继承的方法 us.codecraft.webmagic.downloader.AbstractDownloader

addToCycleRetry, download, download, onError, onSuccess

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

PhantomJSDownloader

public PhantomJSDownloader()

PhantomJSDownloader

public PhantomJSDownloader(String phantomJsCommand)

添加新的构造函数，支持phantomjs自定义命令 example: phantomjs.exe 支持windows环境 phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误 /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException

参数:

phantomJsCommand -

PhantomJSDownloader

public PhantomJSDownloader(String phantomJsCommand,
                           String crawlJsPath)

新增构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js

 crawl.js start --
 
   var system = require('system');
   var url = system.args[1];
   
   var page = require('webpage').create();
   page.settings.loadImages = false;
   page.settings.resourceTimeout = 5000;
   
   page.open(url, function (status) {
       if (status != 'success') {
           console.log("HTTP request failed!");
       } else {
           console.log(page.content);
       }
   
       page.close();
       phantom.exit();
   });
   
 -- crawl.js end
 

具体项目时可以将以上js代码复制下来使用 example: new PhantomJSDownloader("/your/path/phantomjs", "/your/path/crawl.js");

参数:

phantomJsCommand - phantomJsCommand

crawlJsPath - crawlJsPath

方法详细资料

download

public Page download(Request request,
                     Task task)

setThread

public void setThread(int threadNum)

getPage

protected String getPage(Request request)

getRetryNum

public int getRetryNum()

setRetryNum

public PhantomJSDownloader setRetryNum(int retryNum)