incomplete JSON export

Exporting has recently been problematic (here and here) but neither of those reports seemed to indicate problems with the JSON export.

I'm missing the first 600 books I cataloged in my JSON export. There's a few lines of curly brackets to begin the file ("{},},},},},}"), and then the file contains almost all the books I've cataloged since October 21, 2006, but none before. It's missing sixteen books I cataloged on that date as well. Sorting my catalog by date entered shows the first book in the export is exactly book #601 in my catalog.

I use Pale Moon (29.2) as my browser on Win10, but this also occurs on Firefox 88.0.1. I tried to export a few different times yesterday, thinking it was perhaps a "server busy"-type problem, but all the downloads were essentially the same. (I even tried breaking up the export into collections, but that still couldn't find those first 600 books.)

>1 legallypuzzled: Confirming (sort of). My JSON export skips the first 200 books, and starts with book #201. Vivaldi/Win10

Sorry about this - we did not see this in our testing.

We'll circle back and take a look to figure out what's going on there.

Not only am I missing the first 300 books (the fact that we're all missing an even hundred, rather than a date range, suggests the chunker is falling down) but what I do get is malformed json, presumably because it's missing the beginning stuff.

So your testing didn't include "test to make sure the export file has the correct number of records"?

I'm also getting malformed JSON. The file starts with

{
},
},"195926004": {

The rest of the data looks OK from line 3 on. All books with ids less than 195926004 (the first 200 books) are omitted.

Any updates on this?

Oh, yes, I believe that Chris fixed this. I'm going download your export and see if I'm right.

Oh, I can see that Kristi marked these as fixed.

https://www.librarything.com/topic/332011
https://www.librarything.com/topic/332108

Nope. Drat. Still missing some at the beginning. Dropping this on ccatalfo to see what's up.

I'll confirm that it's still happening. In case it helps, my missing ones don't seem to be at the beginning, or at least aren't all at the beginning. I get 2438 of the 3938 it says it exported, and there were several malformed areas (all an extra }, at the beginning of a line) scattered throughout the file.

Hi all - sorry for the delay on this.

It appears that this was an issue with character encoding in some users' data which caused the JSON exports to fail in some cases.

I've switched some code to hopefully handle these better and re-tested each of your exports and I got complete, valid record sets in each case.

Let us know if you still see problems.

>11 ccatalfo: Sorry, it still doesn't appear to work. It starts with a single "empty" entry "{ }," and skips the first 200 books. Interestingly, it skips 200 books whether I'm exporting all books or just my "Physical Books" collection, which means different books get skipped.

ETA: It occurred to me that it might not be a coincidence that it skips 200 books and I'm also displaying 200 books per page on my "Your books" tab. So I changed that setting to display 30 books per page and did another JSON download. No difference; it still skips the first 200 books.

>12 Foretopman: Ok - thanks i'll take a look at yours now.

>12 Foretopman: Ah, sorry, this was a case of long running job not getting the code update.

If you try now, it should work.

(And i've amended the code to restart itself periodically for future code updates.)

Confirming it works; all books are now included. Thanks!

>14 ccatalfo: Thanks, it looks good!

Mine's good now too. Thanks!

>14 ccatalfo: Thanks. And 2xThanks for the explanation!