us.codecraft.webmagic.downloader

接口 Downloader

所有已知实现类:

AbstractDownloader, HttpClientDownloader

public interface Downloader

Downloader is the part that downloads web pages and store in Page object.
Downloader has setThread(int) method because downloader is always the bottleneck of a crawler, there are always some mechanisms such as pooling in downloader, and pool size is related to thread numbers.

从以下版本开始:

0.1.0

作者:

code4crafter@gmail.com

方法概要

限定符和类型	方法和说明
Page	download(Request request, Task task) Downloads web pages and store in Page object.
void	setThread(int threadNum) Tell the downloader how many threads the spider used.

方法详细资料

download

Page download(Request request,
              Task task)

Downloads web pages and store in Page object.

参数:

request - request

task - task

返回:

page

setThread

void setThread(int threadNum)

Tell the downloader how many threads the spider used.

参数:

threadNum - number of threads