us.codecraft.webmagic.model

类 OOSpider<T>

java.lang.Object

us.codecraft.webmagic.Spider

us.codecraft.webmagic.model.OOSpider<T>

所有已实现的接口:

Runnable, Task

public class OOSpider<T>
extends Spider

The spider for page model extractor.
In webmagic, we call a POJO containing extract result as "page model".
You can customize a crawler by write a page model with annotations.
Such as:

  @TargetUrl("http://my.oschina.net/flashsword/blog/\\d+")
  public class OschinaBlog{

       @ExtractBy("//title")
      private String title;

       @ExtractBy(value = "div.BlogContent",type = ExtractBy.Type.Css)
      private String content;

       @ExtractBy(value = "//div[@class='BlogTags']/a/text()", multi = true)
      private List<String> tags;
 }
 

And start the spider by:

   OOSpider.create(Site.me().addStartUrl("http://my.oschina.net/flashsword/blog")
        ,new JsonFilePageModelPipeline(), OschinaBlog.class).run();
 }
 

从以下版本开始:

0.2.0

作者:

code4crafter@gmail.com

嵌套类概要

从类继承的嵌套类/接口 us.codecraft.webmagic.Spider

Spider.Status

字段概要

从类继承的字段 us.codecraft.webmagic.Spider

destroyWhenExit, downloader, executorService, exitWhenComplete, logger, pageProcessor, pipelines, scheduler, site, spawnUrl, startRequests, stat, STAT_INIT, STAT_RUNNING, STAT_STOPPED, threadNum, threadPool, uuid

构造器概要

构造器

限定符	构造器和说明
protected	OOSpider(us.codecraft.webmagic.model.ModelPageProcessor modelPageProcessor)
	OOSpider(PageProcessor pageProcessor)
	OOSpider(Site site, PageModelPipeline pageModelPipeline, Class... pageModels) create a spider

方法概要

限定符和类型	方法和说明
OOSpider	addPageModel(PageModelPipeline pageModelPipeline, Class... pageModels)
static OOSpider	create(Site site, Class... pageModels)
static OOSpider	create(Site site, PageModelPipeline pageModelPipeline, Class... pageModels)
protected CollectorPipeline	getCollectorPipeline()

从类继承的方法 us.codecraft.webmagic.Spider

addPipeline, addRequest, addUrl, checkIfRunning, clearPipeline, close, create, downloader, extractAndAddRequests, get, getAll, getPageCount, getScheduler, getSite, getSpiderListeners, getStartTime, getStatus, getThreadAlive, getUUID, initComponent, isExitWhenComplete, isSpawnUrl, onError, onSuccess, pipeline, processRequest, run, runAsync, scheduler, setDownloader, setEmptySleepTime, setExecutorService, setExitWhenComplete, setPipelines, setScheduler, setSpawnUrl, setSpiderListeners, setUUID, sleep, start, startRequest, startUrls, stop, test, thread, thread

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

OOSpider

protected OOSpider(us.codecraft.webmagic.model.ModelPageProcessor modelPageProcessor)

OOSpider

public OOSpider(PageProcessor pageProcessor)

OOSpider

public OOSpider(Site site,
                PageModelPipeline pageModelPipeline,
                Class... pageModels)

create a spider

参数:

site - site

pageModelPipeline - pageModelPipeline

pageModels - pageModels

方法详细资料

getCollectorPipeline

protected CollectorPipeline getCollectorPipeline()

覆盖:

getCollectorPipeline 在类中 Spider

create

public static OOSpider create(Site site,
                              Class... pageModels)

create

public static OOSpider create(Site site,
                              PageModelPipeline pageModelPipeline,
                              Class... pageModels)

addPageModel

public OOSpider addPageModel(PageModelPipeline pageModelPipeline,
                             Class... pageModels)