Upstream Source Monitoring Guide

Status: ✅ Operational
Last Updated: 2026-07-03
Automation: Daily at 2 AM UTC

🎯 Overview

The upstream source monitoring system automatically fetches updates from trusted blocklist sources and creates pull requests with new domains. This keeps your lists current with the latest security threats without manual intervention.

Key Features

🤖 Fully Automated - Runs daily, creates PRs automatically
📊 Transparent - Every change visible with source attribution
🔒 Trust-Based - Only trusted sources eligible for auto-merge
⚡ Smart Caching - 24-hour TTL reduces bandwidth
📈 Scalable - Add sources by editing YAML config
✅ Quality Control - Manual review for large changes

🏗️ Architecture

Components

Configuration (config/lists.yml) - Defines upstream sources
Monitor Script (scripts/monitor_upstream.py) - Fetches and compares
GitHub Workflow (.github/workflows/upstream-monitor.yml) - Automation
Pull Requests - Generated with detailed change reports

Workflow

┌─────────────────────────────────────────────────────┐
│  Daily Trigger (2 AM UTC)                           │
└─────────────────┬───────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────┐
│  Load config/lists.yml                               │
│  • Find lists with upstream_sources                  │
│  • Get update frequency and trust settings           │
└─────────────────┬───────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────┐
│  For each list with upstream sources:               │
│  1. Fetch upstream data (with caching)              │
│  2. Normalize domains based on format               │
│  3. Compare with local list                         │
│  4. Identify new and removed domains                │
└─────────────────┬───────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────┐
│  If changes found:                                  │
│  1. Create git branch                               │
│  2. Add new domains to list                         │
│  3. Commit changes                                  │
│  4. Generate PR description                         │
└─────────────────┬───────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────┐
│  Create Pull Request:                               │
│  • Add labels (size:small/medium/large)             │
│  • Mark auto-merge eligible if ≤10 domains          │
│  • Request review if >100 domains                   │
└─────────────────┬───────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────┐
│  Create summary issue:                              │
│  • List all updated lists                           │
│  • Total new domains                                │
│  • Links to PRs                                     │
└─────────────────────────────────────────────────────┘

📋 Configuration

Current Lists with Upstream Sources

List	Category	Sources	Frequency
abuse	Security	2	Daily
ads	Advertising	3	Daily/Weekly
crypto	Security	2	Daily/Weekly
fraud	Security	1	Daily
gambling	Content	1	Weekly
malware	Security	3	Daily
phishing	Security	2	Daily
porn	Content	1	Weekly
ransomware	Security	2	Daily
tracking	Privacy	2	Weekly

Total: 23 upstream sources configured

Adding a New Upstream Source

Edit config/lists.yml and add to the appropriate list:

lists:
  your_list:
    description: "Description of your list"
    category: security  # or advertising, content, tracking, etc.
    status: stable
    upstream_sources:
      - url: "https://example.com/blocklist.txt"
        format: hosts  # hosts, domains, adguard, or dnsmasq
        trusted: true  # true = auto-merge eligible, false = always review
        update_frequency: daily  # daily, weekly, hourly
        filter_comments: true  # Skip lines starting with #
        max_domains: 1000  # Optional: limit to prevent huge merges

Format Types

Format	Example	Description
`hosts`	`0.0.0.0 example.com`	Standard hosts file format
`domains`	`example.com`	Plain domain list (one per line)
`adguard`	`\\|\\|example.com^`	AdGuard format
`dnsmasq`	`server=/example.com/`	dnsmasq format

Trust Levels

trusted: true - Auto-merge eligible if changes ≤ threshold
trusted: false - Always requires manual review

Update Frequencies

daily - Checked every day at 2 AM UTC
weekly - Checked once per week
hourly - Checked every hour (use sparingly!)

🚀 Usage

Manual Execution

# Check a specific list
python scripts/monitor_upstream.py --list ads

# Check all lists with upstream sources
python scripts/monitor_upstream.py --all

# Dry run (check only, don't create PRs)
python scripts/monitor_upstream.py --all --dry-run

# Force fresh fetch (ignore cache)
python scripts/monitor_upstream.py --all --no-cache

Automated Execution

The workflow runs automatically via GitHub Actions:

Schedule: Daily at 2 AM UTC
Trigger: .github/workflows/upstream-monitor.yml
Manual: Actions tab → “Upstream Source Monitoring” → Run workflow

Manual Workflow Options

Specific List: Set list_name input (e.g., ads)
All Lists: Leave list_name empty
Dry Run: Set dry_run to true

📊 Pull Request Format

Example PR Title

chore: update malware from upstream sources (+8 domains)

Example PR Body

## 🤖 Automated Upstream Update: malware

This PR was automatically generated by the upstream monitoring system.

### 📊 Summary

- **List:** malware.txt
- **New domains:** 8
- **Removed domains:** 0
- **Sources checked:** 3

### 📡 Source Details

#### Source 1: urlhaus-filter-hosts.txt

- **URL:** https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-hosts.txt
- **Upstream total:** 5,234 domains
- **New domains:** 8

**New domains:**

malicious-domain1.com malicious-domain2.com malicious-domain3.com …

### ✅ Validation

- [ ] New domains are relevant to the malware category
- [ ] No false positives identified
- [ ] Domains pass validation checks
- [ ] Build succeeds

### 🔄 Merge Policy

✅ **Auto-merge eligible** - Changes are below threshold (≤10 domains)

Labels Added

automation - Automated PR
upstream-update - From upstream monitoring
size:small - ≤10 domains
size:medium - 11-100 domains
size:large - >100 domains
auto-merge-candidate - Eligible for auto-merge
needs-review - Requires manual review
breaking-change - Large changes (>100 domains)

🔧 Merge Policy

Automatic Merge Eligibility

Changes are eligible for automatic merge when ALL conditions are met:

✅ Source is trusted (trusted: true in config)
✅ Small changes (≤10 new domains)
✅ CI passes (all tests green)
✅ No conflicts (clean merge)

Manual Review Required

Manual review is required when ANY condition is met:

⚠️ Source not trusted (trusted: false)
⚠️ Medium changes (11-100 domains)
🔴 Large changes (>100 domains)
🔴 CI fails (tests or validation errors)

Review Checklist

When reviewing a PR:

✅ Verify domains are relevant to the list category
✅ Check for false positives (legitimate sites)
✅ Ensure no typos in domain names
✅ Confirm source is reputable
✅ Check CI results (all tests passing)
✅ Review sample domains in PR description

🎯 Thresholds & Limits

Global Settings (config/lists.yml)

settings:
  upstream:
    enabled: true
    check_frequency: daily
    auto_merge_threshold: 10      # Auto-merge if changes ≤ this
    require_review_threshold: 100  # Manual review if changes > this
    cache_ttl: 86400              # 24 hours in seconds

Per-Source Settings

upstream_sources:
  - url: "..."
    max_domains: 1000  # Limit to prevent huge merges
    trusted: true      # Auto-merge eligible

📈 Monitoring & Statistics

Daily Summary Issue

After each run, a summary issue is created with:

Total new domains across all lists
Number of lists updated
Number of PRs created
Links to each PR
Auto-merge eligibility status

Weekly Reports

The weekly issue report (created by weekly-report.yml) includes:

Upstream PRs created
Auto-merge success rate
Manual review turnaround time
Source health statistics

🔍 Troubleshooting

Common Issues

1. Source URL Not Accessible

Symptom: Workflow fails with “Error fetching URL”

Solution:

Check if upstream source is still online
Verify URL is correct in lists.yml
Check for rate limiting or IP blocks
Review GitHub Actions logs

2. Too Many Changes

Symptom: PR marked as breaking-change with 1000+ domains

Solution:

Review upstream source quality
Add max_domains limit to source config
Consider splitting large sources
Verify source format is correct

3. False Positives

Symptom: Legitimate domains in PR

Solution:

Mark source as trusted: false for manual review
Report false positives to upstream source
Add domains to local allowlist
Consider different upstream source

4. Cache Issues

Symptom: Not detecting updates from upstream

Solution:

# Clear cache and force refresh
python scripts/monitor_upstream.py --all --no-cache

# Or delete cache files
rm -f /tmp/upstream_cache_*.txt

5. Workflow Not Running

Symptom: No PRs created at scheduled time

Solution:

Check workflow is enabled in Actions tab
Verify cron schedule in .github/workflows/upstream-monitor.yml
Check GitHub Actions status page
Review workflow run history for errors

🛡️ Security Considerations

Source Validation

Before adding a new upstream source:

✅ Verify reputation - Is the source well-known and trusted?
✅ Check update frequency - How often is it updated?
✅ Review content quality - Any false positives?
✅ Test manually - Run with --dry-run first
✅ Monitor initially - Set trusted: false until proven reliable

Sensitive Sources

For security-critical lists (ransomware, phishing):

Set trusted: false to always require review
Limit max_domains to catch anomalies
Monitor more frequently (daily or hourly)
Set up alerts for large changes

📚 Examples

Example 1: Adding AdGuard DNS Filter

lists:
  ads:
    upstream_sources:
      - url: "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt"
        format: adguard
        trusted: true
        update_frequency: daily
        max_domains: 5000

Example 2: Adding Security List (Requires Review)

lists:
  phishing:
    upstream_sources:
      - url: "https://example.com/new-phishing-list.txt"
        format: domains
        trusted: false  # Require manual review
        update_frequency: daily
        max_domains: 500

Example 3: Multiple Sources for One List

lists:
  malware:
    upstream_sources:
      - url: "https://urlhaus.abuse.ch/downloads/hostfile/"
        format: hosts
        trusted: true
        update_frequency: daily
      
      - url: "https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-hosts.txt"
        format: hosts
        trusted: true
        update_frequency: daily
      
      - url: "https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt"
        format: domains
        trusted: false
        update_frequency: weekly
        max_domains: 1000

📞 Support

Getting Help

Documentation: This file
Issues: GitHub Issues
Workflow Logs: Actions tab → Upstream Source Monitoring
Configuration: config/lists.yml

Reporting Bugs

If upstream monitoring is not working:

Check the workflow run logs
Review recent changes to config/lists.yml
Test manually with --dry-run
Open an issue with logs and details

🎓 Best Practices

✅ DO

Start with trusted: false for new sources
Set reasonable max_domains limits
Monitor PRs in the first week
Use caching (cache_ttl: 86400)
Group related sources in same list
Document source in comments

❌ DON’T

Set all sources to trusted: true immediately
Use hourly frequency unless necessary
Skip manual review of large PRs
Ignore build failures
Add sources without testing
Disable caching globally

🔮 Future Enhancements

Planned Features

VirusTotal Integration - Reputation checks for new domains
Incremental Updates - Track last-seen dates for efficiency
Source Health Monitoring - Detect broken URLs automatically
IP Address Support - Monitor upstream .ip files
Scheduled Health Reports - Weekly source reliability stats
Auto-Remove Dead Sources - Remove consistently failing sources

Feature Requests

Have an idea? Open an issue with the feature-request label!

📊 Statistics

Current Status

Lists Configured: 10
Upstream Sources: 23
Update Frequency: Daily (2 AM UTC)
Average PR Size: ~5-15 domains
Auto-Merge Rate: ~80% (small trusted changes)
Cache Hit Rate: ~90% (during same day)

Performance

Fetch Time: ~5-10 seconds per source (without cache)
Cache Hit Time: <1 second per source
PR Generation: <30 seconds
Total Runtime: ~2-5 minutes for all lists

Last Updated: 2026-07-03
Version: 1.0
Maintainer: Block List Project Team