web crawler

Bibliopedia

Submitted by Michael Widner on

Bibliopedia is an open source, semantic wiki research platform designed to crawl scholarly resources including JSTOR, the Library of Congress, the Arts and Humanities Citation Index, and similar data sources, extract metadata about works cited, convert that data into a semantic web format, aggregate the different repositories, then display the results on a wiki-style website for the scholarly community to verify, add to, annotate, elaborate, and discuss.

Webpage: 

http://www.bibliopedia.org

Tags: 

Heritrix

Submitted by Quinn Dombrowski on

Heritrix is web crawler used by the Internet Archive, which provides a web-based user interface after initial configuration on a Linux machine. Also used by the Library of Congress, Heritrix captures metadata in the Web ARChive (WARC) format.

Webpage: 

http://crawler.archive.org/

Tags: 

Pattern

Submitted by Quinn Dombrowski on

Pattern is a Python web mining module with tools for data retrieval (Google + Twitter + Wikipedia API, web spider, HTML DOM parser), text analysis (rule-based shallow parser, WordNet interface, syntactical + semantical n-gram search algorithm, tf-idf + cosine similarity + LSA metrics) and data visualization (graph networks).

Webpage: 

http://www.clips.ua.ac.be/pages/pattern

Tags: 

PDFmyURL

Submitted by Quinn Dombrowski on

PDFmyURL converts websites to PDFs. Premium, paid features include watermarks and preference configuration (e.g. paper size, table of contents, disabling/enabling links, etc.)

Webpage: 

http://pdfmyurl.com/

Tags: 

Subscribe to web crawler