User-agent: * Disallow: *?* Disallow: /search/ Disallow: /style-assets/ Disallow: /staff/ Disallow: /training/ Disallow: /current-students/calendar/ Disallow: /lib/skills-for-learning/calendar/ Disallow: /courses/search/ Disallow: /ongoing-development/ Disallow: courses/course-editable-foss/ Disallow: /courses/course-check/ Disallow: /news-and-events/open-day/undergraduate-open-day/open-day-dates/ Disallow: /about-us/news-and-events/open-day/ Disallow: /courses/continuing-professional-development-cpd-short-courses/ Disallow: /alumni/get-involved/mentoring/computer-science-mentoring-evaluation/ Disallow: /staff/it-services-disruption-and-recovery-faqs/ Disallow: /news-and-events/wlv-events/ User-agent: funnelback Allow: / crawl-delay: 2 User-agent:Twitterbot Allow: / Allow: /media User-agent: AhrefsBot Disallow: /