Skip to content

J00MZ/crawltricks

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

29 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Objective

Solution to Backend Exercise

Usage

Prerequisites

Running

DEPTH=<DEPTH> URL=<URL> docker-compose up -d
docker-compose logs -f

Outputs

All pages downloaded reside in local directory called <URL_BASE_NAME>.
For example: news.ycombinator.com/

output.tsv file resides in <URL_BASE_NAME> directory
columns: url, depth, ratio

About

Web crawler service

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •