The safety pipeline
Every check tg.pe runs on a link, what it costs, and why it is shaped that way
A shortener lends its own reputation to whatever it points at, so the interesting question is not whether it checks links but when. A check that runs before the link goes live costs the submitter latency; one that runs after costs nothing but lets a bad link exist for a few seconds. tg.pe splits its stages across both, and runs a third pass on later clicks. This page is the whole of it: each stage's data source, what it does on a hit, what happens when the dependency is down, and where the outcome is written.
It is written for anyone running a similar service. None of it needs a team or a budget; most of it is a local list plus one API call. Where a choice is counter-intuitive, a honeypot instead of a refusal, or fail-open on Google Safe Browsing, the design notes say why and name the cost. The methodology page holds the data tables and definitions, and RDAP bootstrap coverage is what the new-domain stage turned out to measure about the DNS.
Where the checks run
tg.pe layers its checks at three points: before a link goes live, after the response has been flushed to the client, and again on every later click. The split is deliberate. The synchronous checks are the ones cheap enough to add to the user-visible latency budget; everything slow runs strictly after the response is flushed, so it adds safety without adding latency anyone feels. A link a background check flags is expired seconds later, live too briefly to circulate, but the submitter never waited on the check. This is the part most directly reusable by another small operator.
Stage-by-stage
Grouped by where each stage runs. Fail-mode is what happens when the external dependency is unavailable.
| Stage | Data source | Blocking? | On hit | Fail-mode | Logged to |
|---|---|---|---|---|---|
| Before the link goes live (synchronous, adds latency) | |||||
| Open-redirector unwrap | Local list of redirectors that carry their target in plaintext | Not blocking | youtube.com/redirect?q= and google.com/url?q= are peeled off, so the checks below read the real destination instead of a trusted host. |
n/a (local) | not logged separately |
| Domain block list & high-risk list | Local curated lists | Blocking | Block list (55): refuse on both channels. High-risk (370): refuse on Web only; identified Telegram users pass. | n/a (local) | submission_rejects |
| URL-shortener shape | Local pattern match | Blocking | A short two-label host with a single short path segment reads as another shortener → refuse, Web only, and only for anonymous visitors outside Taiwan. Signing in or using the bot passes. Exempt: short domains that are not shorteners (t.me, telegra.ph, vk.com, mega.nz, ibb.co) and redirectors that only reach their own operator's property (amzn.to, pin.it, b23.tv, lin.ee, g.co, wa.me). A general-purpose shortener is refused even when a brand runs it. |
n/a (local) | submission_rejects |
| Newly-registered-domain check | RDAP (via rdap.org); CIRCL pDNS second tier | Blocking | Web < 30 days / Telegram < 7 days → five-minute honeypot, operator notified. ccTLD with no RDAP falls to pDNS first-seen. | Fail-open (null age is never treated as "new") | submission_rejects |
| Ad-smartlink pattern | Local pattern match | Blocking | CPM/affiliate smartlink shape (ad-network hosts, opaque key params, unfilled affiliate macros) → five-minute honeypot, flagged. | n/a (local) | submission_rejects |
| Google Safe Browsing: at submission | Google Safe Browsing API | Blocking | Threat-category match → refuse before the short link is created. | Fail-open: API error lets it through, logged | error log + submission_rejects |
| Submitter rate limit | Local counters | Blocking | Two-layer: Web 5/hr (anon non-TW) up to 20/hr (identified) plus a 10-minute burst cap; Telegram 60/hr plus a 20-per-10-min burst cap → refuse. The burst layer is what stops the cron-bot spikes. | n/a (local) | rate_limit_hits |
| After the response is flushed (background, no user-visible latency) | |||||
| VirusTotal reputation + creation date | VirusTotal v3 /domains | Post-flush | Malicious verdict (≥ VT_MALICIOUS_MIN = 2 engines) or brand-new on an RDAP-missed TLD → tighten link to 30-min expiry. Whitelist: telegra.ph, graph.org, t.me (Telegram-owned, abused on sub-paths only) are never flagged. |
Fail-open (any VT failure / rate-limit leaves the link untouched) | malicious_detections |
| urlscan.io capture | urlscan.io | Post-flush | New-domain / VT honeypots get a screenshot, redirect chain, contacted hosts. Runs on urlscan's infrastructure → hostile page never fetched from tg.pe's server. Becomes evidence for onward reports. | Fail-open (no capture, no penalty) | capture stored with the link |
| Submitter IP screen | AbuseIPDB | Post-flush | New non-Taiwan Web submissions scored; AbuseIPDB confidence score > 75 → link expired within a second or two of creation. Never runs for Telegram. | Fail-open | error log |
| Google Safe Browsing: re-check on later clicks | Google Safe Browsing API | Post-flush (after redirect sent) | Only if last check > 7 days; atomic claim so concurrent clicks check once; flagged → removed, operator notified. | Fail-open: next click re-checks | error log |
| Always on | |||||
| Manual abuse intake | abuse@tg.pe and /abuse | n/a | Triaged by hand; the verified median from external report to removal is ~4 hours (one stranger-reported case closed in 50 minutes). | n/a | mail archive |
| Machine-readable security contact | security.txt (RFC 9116); OpenPGP via WKD |
n/a | Kept current; enables encrypted reports. | n/a | n/a |
Design notes: why it works this way
Why it works this way (design rationale)
- Honeypot, not hard refusal. A brand-new-domain or smartlink submission is created with a five-minute expiry rather than refused; the submitter sees success while the operator is notified at once. A hard refusal would hand the adversary a clean feedback signal ("this destination is flagged, rotate it") which the honeypot deliberately denies.
- urlscan.io scopes the SSRF claim precisely. The capture runs on urlscan's own infrastructure, so a hostile page is never fetched from tg.pe's server. This is not a claim that tg.pe never contacts destinations. RDAP and VirusTotal lookups are made by tg.pe directly; the precise claim is only that the page-rendering fetch of a flagged hostile destination is offloaded.
- Fail-open Safe Browsing is a tradeoff, and an attack surface. Letting a submission through on a GSB error beats blocking legitimate users on Google's downtime. The cost is real: a GSB outage is a window, and it is named here as a known limitation rather than mitigated away.
- Re-check fires on exposure, not on a schedule. Re-scanning ~30,000 idle URLs daily would burn API quota for nothing; the re-check instead fires the moment someone is about to be exposed, and only if the last check is > 7 days old. The atomic claim in the table is what stops a burst of concurrent clicks turning that into one call per click.
- Asymmetric trust. Web is anonymous and mutable, so it gets the stricter thresholds (30-day new-domain window, tighter rate limits, no custom codes, the AbuseIPDB screen). Telegram identity is durable and bannable, so it is trusted further (7-day window, higher rate limit, high-risk list bypassed).
What the automation catches
The logging is built to measure the defence funnel, not just to fire alerts. EventLog::logReject() writes one row per failing stage into submission_rejects, by design, so the operator can measure how much each stage caught on its own versus in overlap across blacklist / new_domain / safe_browsing / abuseipdb; and malicious_detections records the source, reason, and engine counts for each background tightening.
One slice is reportable now. Of the background VirusTotal scans run since that pass went live, 148 tightened a just-created link to a 30-minute expiry with no operator in the loop, split below.
| Malicious reputation verdict | 120 |
| Brand-new domain on an RDAP-missed TLD | 28 |
Some of these 148 would have been caught by another stage too, so nothing here says every layer is independently essential. The per-stage funnel that would settle which stages are redundant and which are load-bearing is still open.
Safety-mechanism timeline
The change history, paired where relevant with the miss each fix responded to.
CIRCL MISP outbound publication operational (taxonomy-tagged community events); urlscan.io evidence capture; the AbuseIPDB IP screen and the click-time Safe Browsing re-check moved to post-response background passes (expire-on-flag) to keep latency off the hot path; first externally-confirmed closed-loop takedown.
Smartlink-pattern detection; CIRCL Passive DNS online as the RDAP second tier; OpenPGP key published via Web Key Directory; the outbound contributor pipeline went operational.
Google Safe Browsing at submission (fail-open); soft-delete + 7-day click-driven re-check; public /abuse page.
RDAP new-domain rejection; RFC 9116 security.txt; Telegram /rm takedown command. Cross-vendor: false-positive reclassifications (Forcepoint, alphaMountain, BrightCloud, Chống Lừa Đảo), VirusTotal / Trend Micro verification, TWCERT/CC + abuse.ch / Spamhaus + CIRCL Passive DNS onboarding.
High-risk domains reframed to "Telegram-only"; ASN logic simplified.
Removed links return HTTP 404 instead of redirecting (no metadata leak).
ASN-level banning of commercial VPN/VPS ranges, the direct response to the IRS + CIRCL bulk wave.
IP block list + AbuseIPDB pre-screen (after the Orange France run); free-TLD ban; block list extended to Telegram.
Launch, fbclid stripping, bot-UA shortcut, regex-validated codes; Web rate limit.