Web and SEO tools.

Decide per kind. Training crawlers (GPTBot, ClaudeBot, CCBot) and the control tokens Google-Extended and Applebot-Extended can go without touching search rankings. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are how those assistants cite you, and the user fetchers (ChatGPT-User, Claude-User) read a page someone asked about. The robots.txt generator has presets for training only and for all of them.

No. robots.txt controls crawling: a disallowed URL can still be indexed from links pointing at it, shown without a description. A noindex meta tag or X-Robots-Tag header keeps it out, and Google can only see that on a page it is allowed to crawl.

It helps search engines find URLs, especially on large or new sites, and an accurate lastmod tells them what changed. It doesn't make a page indexed: Google still judges each page on its own.

Both at the root of the host: https://example.com/robots.txt and, usually, https://example.com/sitemap.xml. A Sitemap line in robots.txt points every crawler at the sitemap, and Search Console and Bing Webmaster Tools take it directly.

base-uri, form-action and frame-ancestors don't fall back to default-src, so a policy of default-src 'self' alone still lets an injected <base> or form go anywhere and any site frame the page. Set them, plus object-src 'none', in every policy.

report-uri is the original: a URL the browser POSTs violation reports to, still the only one Firefox uses. report-to names an endpoint declared in a Reporting-Endpoints header and sends reports through the Reporting API. Send both: browsers that know report-to ignore report-uri.

frame-ancestors supersedes it, and browsers that support frame-ancestors ignore X-Frame-Options when both are sent. frame-ancestors can also list several allowed parents, which X-Frame-Options can't. Keep X-Frame-Options only for very old browsers.

event.key for what the key means (Enter, Escape, a typed letter) and event.code for where it is (WASD movement, shortcuts that should survive layouts). keyCode and which are deprecated, differ between browsers and layouts, and read 229 during IME composition.

favicon.ico with 16, 32 and 48 pixel frames, favicon.svg for letters, emoji and SVG logos, a 180-pixel apple-touch-icon.png, 192 and 512 pixel PNGs, a maskable 512 and site.webmanifest, with the link tags and the Next.js app/ files, all in one ZIP.

Yes. Windows shortcuts, older browsers and most crawlers and feed readers ask for /favicon.ico at the site's root and read nothing else. Keep both: browsers that read SVG icons take favicon.svg, and everything else gets the ICO.

A manifest with a name, a start URL, a standalone, fullscreen or minimal-ui display and 192 and 512 pixel icons, served over HTTPS. The web app manifest generator checks each of these as you type and says what's missing.

The central circle, 80% of the icon's width, that Android shows whatever shape its launcher crops to. Both icon tools draw the maskable icon with its content inside that circle and outline it in the preview.

Meta tags in the head describe a page to search engines: the title, the description, robots and the canonical link. Open Graph is Facebook's set of og: tags for share cards, read by nearly every app that unfurls a link. Twitter cards are X's twitter: tags, which fall back to Open Graph; the meta tag generator writes all three at once.

About 600 px on a desktop, which is 50 to 60 characters of 20px Arial depending on the letters: wide ones such as M and W use it up faster than i and l. The meta tag generator measures the real width as you type and shows where Google cuts.

1200 × 630 px, a 1.91:1 ratio, as PNG, JPEG or WebP under 5 MB. It shows large on Facebook, LinkedIn, Slack and Discord, and X crops it to 2:1. No platform shows an SVG.

No. They are drawn from the fields you fill in, so they work before the page is live and send it nowhere; only an image URL you paste is loaded, by your browser, to show it.

One digit each for the owner, the group and everyone else, each the sum of read (4), write (2) and execute (1). So 750 is everything for the owner, read and execute for the group and nothing for others. A fourth digit in front, when present, holds setuid (4), setgid (2) and the sticky bit (1).

600, or 400 for one you never edit: OpenSSH refuses a private key that the group or others can read, with an UNPROTECTED PRIVATE KEY FILE warning. The ~/.ssh directory itself should be 700 and authorized_keys 600.

The server runs as its own user (www-data, nginx), so it reaches your files as other or as the group. It needs read on files (644) and execute on every directory on the way (755); write belongs only to the account that deploys.

Not for ranking: Google ignores them as signals. But a tagged URL is a different URL, so a page reachable with and without tags can be crawled twice. Point a rel=canonical at the clean URL, and keep UTM tags out of internal links and sitemaps.

The host has non-ASCII letters, which DNS carries as Punycode (xn-- labels). Browsers show the Unicode name unless it mixes scripts in a way that could imitate another domain, a homograph attack, and then they show the xn-- form to warn you. The URL parser shows both forms.

Not gone, but frozen. Chrome finished its user agent reduction in 2023: the version reads 0.0.0 after the major number, and the Windows, macOS and Android details are fixed values. The details moved to Client Hints, which Safari and Firefox don't send.

When it says so: GPTBot, ClaudeBot, PerplexityBot, CCBot and the others name themselves, and the user agent parser recognises them. A scraper can borrow any name, though, so compare the IP with the ranges the operator publishes before you trust it, and block crawlers in robots.txt by the same tokens.