us.codecraft.webmagic.scheduler

类 BloomFilterDuplicateRemover

java.lang.Object

us.codecraft.webmagic.scheduler.BloomFilterDuplicateRemover

所有已实现的接口:

DuplicateRemover

public class BloomFilterDuplicateRemover
extends Object
implements DuplicateRemover

BloomFilterDuplicateRemover for huge number of urls.

从以下版本开始:

0.5.1

作者:

code4crafer@gmail.com

构造器概要

构造器

构造器和说明
BloomFilterDuplicateRemover(int expectedInsertions)
BloomFilterDuplicateRemover(int expectedInsertions, double fpp)

方法概要

限定符和类型	方法和说明
int	getTotalRequestsCount(Task task)
protected String	getUrl(Request request)
boolean	isDuplicate(Request request, Task task)
protected com.google.common.hash.BloomFilter<CharSequence>	rebuildBloomFilter()
void	resetDuplicateCheck(Task task)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions)

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions,
                                   double fpp)

参数:

expectedInsertions - the number of expected insertions to the constructed

fpp - the desired false positive probability (must be positive and less than 1.0)

方法详细资料

rebuildBloomFilter

protected com.google.common.hash.BloomFilter<CharSequence> rebuildBloomFilter()

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

指定者:

isDuplicate 在接口中 DuplicateRemover

getUrl

protected String getUrl(Request request)

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

指定者:

resetDuplicateCheck 在接口中 DuplicateRemover

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

指定者:

getTotalRequestsCount 在接口中 DuplicateRemover