Any good web crawler has limits.

Yeah. Like, literally just:

• Keep track of which URLs you’ve been to
• Avoid going back to the same URL
• Set a soft limit, once you’ve hit it, start comparing the contents of the page with the previous one (to avoid things like dynamic URLs taking you to the same content)
• Set a hard limit, once you hit it, leave the domain altogether

What kind of lazy-ass crawler doesn’t even do that?

And the biggest issue for me is the compensation and management side of things. I have no idea how to get hired as a manager because I spent all of my interviews in the past as a software engineer. And my software engineer skills have basically disappeared over the last 4+ years of managing.

Honestly, you should just apply for the position and see if you get through the interview (you don’t need to quit your current job before accepting interviews). A few of my friends at work basically went from software developers into management. A lot of places actually look for ex-devs for their management because the experience carries over.

I’m in a similar position right now where the dev team I’m working in is making me absolutely miserable, and I just brushed-up the resume and started looking for another job.

Hope you pull through! Toxic teams really suck!