Do you use Common Lisp? Do you scrape web pages, and need to deal with the robots.txt file to play nicely?
Well have I got a deal for you! For the low, low price of absolutely free you too can parse robots.txt files with my Common Lisp library machina-policy1. See the readme for a brief overview.
Don't forget to join the mailing list to offer ideas, submit bugs, and make requests for improvement.
For more web-scraping fun, you might try some libraries with which I am completely unaffiliated and haven't yet tried: cl-web-crawler, cl-kappa, and css-selectors; or my own Oh, Ducks!.
- machina-policy was very briefly named robots.txt. How not to name a library, ladies and gentlemen.