Finding Dead Websites

As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change.

This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and parking.

Table Of Contents

• Feature Rationale
• Data Representation
  • Live Data
  • Event Data
• Change Detection Details
  • Availability Detection
  • Ownership Changes
  • DNS
• Implementation Hurdles
  • Scheduling
  • Certificate Validation
• Conclusions