+- Drunkard's Walk Forums (http://www.accessdenied-rms.net/forums)
+-- Forum: General (http://www.accessdenied-rms.net/forums/forumdisplay.php?fid=1)
+--- Forum: Forums (http://www.accessdenied-rms.net/forums/forumdisplay.php?fid=3)
+--- Thread: One forum left. (/showthread.php?tid=8565)
One forum left. - Bob Schroeck - 09-11-2017
<yawns> I was up about two hours later than I should have been last night, getting the last of "Other People's Fanfiction" from the initial scrape loaded. Between dealing with about 30 or so threads that failed because of the string concatenation problem (requiring me to back out the failed thread, find the missing content, reconstruct the insert statement, and re-run, followed in a few cases by repeating the process because I introduced a new error) and getting caught up in reading some of the oldest content when I should have been paying attention to the DB tasks, it took me a good 8 hours or so.
"General Chatter" has something like three times as many threads as "OPF" does. I'm warning people now that it's not going to happen in one night. (One good thing is that as far as I can recall, "General Chatter" is unlikely to have many posts in excess of 43.5K in size -- the point at which that string problem kicks in. As a result, I should have an overall faster throughput on the transfer. Fixing the failed inserts is the largest time overhead in the process.)
Oh, also, fair warning, folks: to minimize lost threads/posts, I'm going to lock the forum before I start the move (which is really how I should have done it for the other fora, but oh well...). So basically starting tonight, everything is going to be closed over on the old boards.
RE: One forum left. - robkelk - 09-11-2017
"Hey, laughing boy! One fowwum left!"
"I know, I know..."
RE: One forum left. - Bob Schroeck - 09-11-2017
Heh.
Aw, man, that's how it all should've gone. - Bob Schroeck - 09-11-2017
3800 threads. 39,000 posts.
25 minutes.
Ladies and gentlemen, Tapatalk has left the building.
RE: One forum left. - LynnInDenver - 09-11-2017
And I've now deleted the old bookmark. And there was much rejoicing.
(yay)
RE: One forum left. - Bob Schroeck - 09-11-2017
Okay, so I've got one last round of inserts to do, to bring in the incremental scrape of the last month's stuff. And I need to go through the db and make sure everyone's posts are properly credited to them (not as big a job as it sounds, honestly).
But you know, I think I'm going to rest on my laurels until tomorrow night.
RE: One forum left. - robkelk - 09-11-2017
I will wait until you've done that, then copy over anything that I posted to the old board that wasn't in the incremental scrape and should be copied over.
RE: One forum left. - Bob Schroeck - 09-12-2017
Well, I also have Brent's Taparip utility. If I can figure out how to give it a lower date boundary for what it pulls in (not impossible, just difficult since I don't know Perl), I can do a final scrape of the last few dozen posts added since he made the incremental. And now that the old boards are completely locked down, I'm no longer racing against time for that.
I just wish we could recover the first few threads posted here, dammit.