User-agent: panscient.com Disallow: / # All robots will spider the domain User-agent: Mediapartners-Google* Disallow: User-agent: * Disallow: /cl/sid.php?f_a=fd&f_c_id=* Disallow: /cl/sid.php?f_a=wl&f_c_id=* Disallow: /companies/view/* Disallow: /companies/notepad Disallow: /companies/to_notepad/* Disallow: /companies/cl_notepad Disallow: /companies/weblink/* # Disallow directory User-agent: * Disallow: banner User-agent: * Disallow: logos User-agent: * Disallow: css User-agent: * Disallow: images