So, after after about two weeks of the bot running nonstop, things began to choke. I'm not sure if it was the site being slow to serve pages, the new layout messing with things, or simply the code not scaling well with the size and design.
It got a little bit further, but I've come to the conclusion that while this solution may, technically, work, it will not be able to deliver on any reasonable timescale. Especially considering the further conditioning you'll need to extract the data and dump it back into a database. I'll toss along what I've managed to grab, but we'd probably be better served looking at other options.
---
The Master said: "It is all in vain! I have never yet seen a man who can perceive his own faults and bring the charge home against himself."
>Analects: Book V, Chaper XXVI
It got a little bit further, but I've come to the conclusion that while this solution may, technically, work, it will not be able to deliver on any reasonable timescale. Especially considering the further conditioning you'll need to extract the data and dump it back into a database. I'll toss along what I've managed to grab, but we'd probably be better served looking at other options.
---
The Master said: "It is all in vain! I have never yet seen a man who can perceive his own faults and bring the charge home against himself."
>Analects: Book V, Chaper XXVI