Received: from mail.netlandish.com (mail.netlandish.com [174.136.98.166])
by code.netlandish.com (Postfix) with ESMTP id 43F7627B
for <~netlandish/links-dev@lists.code.netlandish.com>; Fri, 25 Apr 2025 00:47:06 +0000 (UTC)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.52; helo=mail-ua1-f52.google.com; envelope-from=peter@netlandish.com; receiver=<UNKNOWN>
Authentication-Results: mail.netlandish.com;
dkim=pass (1024-bit key; unprotected) header.d=netlandish.com header.i=@netlandish.com header.b=AnnKHplO
Received: from mail-ua1-f52.google.com (mail-ua1-f52.google.com [209.85.222.52])
by mail.netlandish.com (Postfix) with ESMTP id A78971D6463
for <~netlandish/links-dev@lists.code.netlandish.com>; Fri, 25 Apr 2025 00:47:17 +0000 (UTC)
Received: by mail-ua1-f52.google.com with SMTP id a1e0cc1a2514c-86feb848764so657407241.0
for <~netlandish/links-dev@lists.code.netlandish.com>; Thu, 24 Apr 2025 17:47:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=netlandish.com; s=google; t=1745542037; x=1746146837; darn=lists.code.netlandish.com;
h=content-transfer-encoding:mime-version:message-id:date:subject:cc
:to:from:from:to:cc:subject:date:message-id:reply-to;
bh=4aPpuYgQ5ztZmSLXnP80OBxAb4MvXppO5orS5zr0/mw=;
b=AnnKHplOd3oXZVDUh2xZng+Fv0k5Hg4XDzKk3yLgibZR0l9esaOM48KKw2URgBbGWu
cuy64mN7l74HRSmPmKAIdYG9yJtHN/mIo8uPUoUPEIt46xv9WkBJSLAuIloPQDYvPHL/
Lq1ZTNb/yaZ1ckUurmFAdT7iAipc7BSI/6zcw=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1745542037; x=1746146837;
h=content-transfer-encoding:mime-version:message-id:date:subject:cc
:to:from:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=4aPpuYgQ5ztZmSLXnP80OBxAb4MvXppO5orS5zr0/mw=;
b=JY2i7dLlcedDwJ7ARU5szP56x3pLsE3P7czjqWRjzSNoEzhkuRiyDzUMyi+FdFdSA9
PAnY09mq+LpVOJZ2EOT2GEv7G/qXgq3NC9Gi883xxeVImXNhXgFS53TXEp1TLcllPOnK
FxbdwQJ0/tYnOLvBAk/HwzPpFhyi9brYfXM7fV2VUUHyc/DwPAWhReLhYVKJ7OLZ+ulN
EzmJowA4JMUElNhwh7EeT4avpFSJCLP2Sn4Kp1ayFms6CDdB88cfgfY8p4gf/PGC72fM
Es+LdEevQVQoJzJcYULr4Q6NxhhpjBEJLUb4A1aX3/2spOR9g0LBynIVyzB5XoRJnMAK
DJnA==
X-Gm-Message-State: AOJu0Yx/QdeFUyrxTP18ULQpINzPwSvvu90FZm2umDWBMRLqSOhH27vg
13mtp1fmeeqByzPpAt9Oiz6LnKYEaIH6H+zqYz7hbHdFyG3wwEcya0JmZvo9Egar/8qWmQ8VctP
0JNQ=
X-Gm-Gg: ASbGncudZem8Cfzs8BocdXpe8yLlVgG+8hIx4Ma5bJb410PZTK0OR+E7VXzFoRkcRI9
SRzIiPvWVSScsDfaF55CRt7cxfdloD3ppUKCdeW3nf43vRyGiNgoIFP/RLLT388Vube8ZFUUo2E
gyLlDq8reVRaCh1vX2Mgf1k5OkdjjWCDQ7pmQeV2q0mC7TgixAJtNXEO8HZV7/gZR4jGJToQh6d
MMaCD7EJZWbyA2gP2kV+FFncxWoXYh1mDS7nvZPOGSzThn3+rCMDDL6FnnpftQWGclElp7VIjzS
1MI2nmZnTs1iqkmLlu0lJcCU28x4pWtddMFjTX7B
X-Google-Smtp-Source: AGHT+IGHNNmI9DNbScYySIadksOdoY1u+Zbh/SDtAw+knnf3LIEBZsdAZeUNfwDpNLxWYGEJDEgeLQ==
X-Received: by 2002:a05:6102:1495:b0:4c3:9b0:9e6b with SMTP id ada2fe7eead31-4d5437d7059mr99771137.10.1745542036709;
Thu, 24 Apr 2025 17:47:16 -0700 (PDT)
Received: from localhost ([2803:2d60:1118:5ee:f7b:3257:a38e:1e9e])
by smtp.gmail.com with ESMTPSA id a1e0cc1a2514c-877aee251ccsm489290241.12.2025.04.24.17.47.16
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Thu, 24 Apr 2025 17:47:16 -0700 (PDT)
From: Peter Sanchez <peter@netlandish.com>
To: ~netlandish/links-dev@lists.code.netlandish.com
Cc: Peter Sanchez <peter@netlandish.com>
Subject: [PATCH links] Adding robots.txt, security.txt and some robots meta headers to help with the crawling activity.
Date: Thu, 24 Apr 2025 18:47:11 -0600
Message-ID: <20250425004713.12560-1-peter@netlandish.com>
X-Mailer: git-send-email 2.47.2
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
---
static/robots.txt | 8 ++++++++
static/security.txt | 11 +++++++++++
templates/link_list.html | 3 +++
3 files changed, 22 insertions(+)
create mode 100644 static/robots.txt
create mode 100644 static/security.txt
diff --git a/static/robots.txt b/static/robots.txt
new file mode 100644
index 0000000..c4750a2
--- /dev/null
+++ b/static/robots.txt
@@ -0,0 +1,8 @@
+User-agent: *
+Disallow: /
+Allow: /$
+Allow: /blog
+Allow: /blog/
+Allow: /pricing
+Allow: /docs/
+
diff --git a/static/security.txt b/static/security.txt
new file mode 100644
index 0000000..69c3bec
--- /dev/null
+++ b/static/security.txt
@@ -0,0 +1,11 @@
+Contact: peter@netlandish.com
+Encryption (PGP/GnuPG): 90956225167A28F5A0B461A2838FFCD2312235D1
+Encryption Key URL: https://petersanchez.com/publickey.txt
+Alternative email: ~netlandish/security@lists.code.netlandish.com (Admin only,
+private mailing list. You can send encrypted email there as well using the key
+above to encrypt the contents)
+
+SPF/DMARC: Please do not submit reports related to our SPF or DMARC DNS
+entries. We will not pay bounties for these reports. Our entries are set
+correctly for our current needs.
+
diff --git a/templates/link_list.html b/templates/link_list.html
index 838a3eb..a019bbd 100644
--- a/templates/link_list.html
+++ b/templates/link_list.html
@@ -1,5 +1,8 @@
{{template "base" .}}
{{ define "title" }}{{ .title }}{{ end }}
+{{ define "extrahead" }}
+<meta name="robots" content="noindex, follow">
+{{ end }}
{{ if (not .bookmarkListing) }}
<form method="GET"
action="{{ if .isUserDomain }}/{{ else }}{{if .isPopular}}{{reverse "core:popular_link_list"}}{{else if .isOrgLink}}{{.currURL}}{{else}}{{reverse "core:recent_link_list"}}{{end}}{{ end }}{{if .hasUnreadFilter}}?filter=unread{{else if .hasStarredFilter}}?filter=starred{{end}}"
--
2.47.2