A few months ago I gave a tech talk on "SEO for Engineers" for our software developers. We (the SEO team) had found that many engineers didn’t know too much about Search Engine Optimization, and sometimes designed our consumer-facing pages in a way that wasn’t ideal for Top Source Media agency SEO. So we thought it’d be useful to go over the basics of good SEO technique for the larger team … ’re sharing that information with you. There is a lot published about SEO, but most of it is written by SEO agencies - this article is written by engineers, for engineers. Even with the smaller screens in 2012, we can see 3 non-ad results without scrolling. In 2019 we see four ads, a bunch of Google Knowledge Graph items and "people also ask" answers, and then a single real search result! It’s getting harder and harder to even appear on the results page without scrolling, arguably making good SEO even more important. SEO stands for "search engine optimization".

It’s a fancy term for doing things that make your pages rank higher in searches, with the goal of getting more and better traffic from search engines. Because Google is the main search provider, this really means ranking higher in Google searches - but most of the things you should do for Google are good for all search engines. Note that SEO is not SEM, "search engine marketing", which is about paying for keyword-based ads. You can think of the ‘O’ as meaning organic or unpaid search, and the ‘M’ for money or paid search. We won’t cover SEM in this post, but will focus on SEO, particularly how to be indexed and rank well. Some aspects of SEO are fairly straight-forward for engineers (most of the technical stuff), but other aspects are a lot more difficult (actually producing and publishing good content). Here we’ll focus mainly on the straight-forward, technical optimizations you should be doing. The absolute minimum your pages need is to be crawl-able and indexed by the Googlebot, and it’s surprisingly easy to get this wrong.

Let’s go over each of these in a bit more detail. Most medium to large sites have a text file located at /robots.txt that tells "good bots", for example the Google web crawler, what pages they should or shouldn’t crawl. 1. Here’s where the XML sitemap for this site lives. 2. For all user agents (i.e., all crawlers), don’t crawl any URLs starting with /api/ or /staff/. But you’ve got to be a bit careful - for example, what if a well-meaning engineer (who isn’t up on SEO or hasn’t read this article) decides to "block all those annoying bots" and adds "Disallow: /" to the file? Or what if your CEO asks a developer to "create a page about me", and the developer, without knowing about the robots.txt rule, decided to put it at /staff/billg? They won’t realize that it has no chance of being indexed. This kind of thing happens more often than you think. Some companies use a robots.txt parsing library to add automated checks that ensure their important pages are always crawlable.

At the least, you should put robots.txt in Top Source Media digital marketing control and have thorough review on all changes. Obviously pages that you want crawled should be allowed by robots.txt. But you should actively disallow URLs that you don’t need crawled. For example, JSON API responses should not be crawled - Google only allocates a certain amount of crawl budget for your site, so you don’t want it crawling stuff it doesn’t need to. Note that disallowing pages in robots.txt is not a security measure. Any bad bot or hacker can still crawl them. So, for example, /api/ requests disallowed by robots.txt still need to have secure authentication if the information behind them isn’t public. If you’re working with existing web apps, the URL structure may already be in place. One page, one URL: always serve a single page from a single, canonical URL. URL. For example, if you’re serving a page at /staff/billg, don’t also serve the content at /staff/gates/bill.