oh my god WHAT

[edenfalling]

Well, fuck.

I tried to import all my LJ comments, but in order to do that you have to import all your entries as well... and now every single journal entry I made before May 3rd, 2009 (barring the directory posts that didn't exist on LJ in the first place) has been duplicated.

*infinite headdesking ensues*

I guess I will try opening a support ticket, but I kind of doubt there's anything to be done except manually delete SIX YEARS of duplicated posts, and who on earth has time for THAT?

*headdesking continues*

Comments

[rahirah]

Yikes, I've imported over old imports a million times, and never had that happen.

[edenfalling]

It was very unnerving!

[synecdochic]

Saw this on /latest (this is the personal account of denise and I'm on my phone and too lazy to swap to the other account, heh). Fear not! We had enough cases of people with old imports getting duplicates on a re import that we have a way for you to fix it yourself. Either you can delete all your imported entries and re import -- https://www.dreamwidth.org/support/faqbrowse?faqid=255 -- or you can open a support request in the Account Payments category, which is the one that goes straight to me, and I'll run the magic deduplicator script. Do the first if you still have all your previously imported entries on LJ available for re import and you know nobody commented on the DW version of the imported entries from before. Do the second if you aren't sure you still have everything in lj for a re import, or if people have made comments on the DW versions since you first imported. (The de duplicator is not incredibly smart and may wind up with a few dupes left over that it doesn't catch, but it tries really hard, and it will leave any dupes that have DW-native comments.)

[rahirah]

Pardon my barging in, but I was wondering -- I have a links community I manage on both LJ and DW. Since there are no community crossposts (or at least if there are I hadn't heard about them) we've been, for the last umpty years, doing one daily post to LJ, and one to DW. The community on LJ goes back many years before the DW, and I'm in the process of importing the old entries. I was assuming that since the LJ entries aren't crossposts, I'd just have to live with many years of duplicate entries, but would the dedupe script work in a case like that?

[synecdochic]

Yeah, in situations like that there's no real good answer. (The deduplicator script *might* work but i'm leaning towards no, and wouldn't want to try it on a comm like yours without testing it on real data first; no matter how well something is programmed and with how many fall safes, real world data is always way messier than you expect.) I'm guessing that it wouldn't help you, as it's specifically looking at things with the "i was imported" database flag for cocomparisons...

My advice would be: make a communityname_from_lj or communityname_archived or whatever account, import the lj comm into it, link it from the profile of the DW comm. Not perfect, but at least avoids the immediate hassle of duplication.

[tephra]

So I'm going to assume that since I was using Semagic to post to my journal here, on LJ, and on IJ that the magic dedup thing won't work for that case either.

I've decided to import from LJ anyway and will manually delete duplicates with no comments. That should take care of a lot of them but I was hoping there was some way to merge the remainders. I've been considering making a suggestion for an admin console command to do it, give it the entry numbers of duplicate posts, have it put all the comments on one of them and delete the other. Ideally with some form of check that the posts have identical content. I have no idea if that's even feasible though.

[synecdochic]

*wince* yeah, if you used semagic to xpost, there's pretty much nothing that can be done; there's no association between the two. Sorry. :/

It would be theoretically possible to do something like an admin console command to merge entries, but that's theoretically possible in the " It would be difficult to program properly, plus, it would run the risk of massive fuckups if people made a mistake that would require a lot of detangling and a lot of staff attention and risk permanent data loss and/or corruption and/or massive privacy leaks in the worst case scenario". (Like, if the comments from one entry wound up on a different entry, and one was locked and filtered and the other was public, it could get ugly.)

Unfortunately we are a Very Tiny Company without the resources to spend that much staff time on something, especially since it's a problem that isn't a core fundamental part of the site. (Like, don't get me wrong, importing is totally very important, I have sixteen years of my personal shit that I moved over myself! But it's not something people do every time they log in to DW, and so it's not as much of a priority.

[tephra]

That's pretty much what I suspected on the tech issues, and for the limited utility it's really not worth the expense. My journals aren't that busy anyway, especially in recent years where my spoon supply is low, so the duplicates with comments that remain won't be much of a bother. Thanks for replying. :)

[synecdochic]

Ugh, I hear you on the spoon deficit thing. I massively overdid it last week and spent the first few days of this week lying in bed and ignoring the internet (except to read fic, heh) to try and recover in hopes my pain levels would drop a bit. "Oh," I thought to myself, "DW can get along without me for a few days, it's not like anything massive is going to happen." ahahahahaha my timing is impeccable thanks LJ

[rahirah]

Ah, that's pretty much as I feared, but I thought it was worth asking just in case. Thanks!

[synecdochic]

I wish I had a better answer for you!

[edenfalling]

Thank you so much for the advice!

I am not sure whether people have commented on my old imported entries, but deleting them wholesale would wreak havoc with the links in my fic directories and I want to reconstruct those posts even less than I want to manually delete six years of duplicate entries.

Will the de-duplicator get rid of the initial import or the more recent import? Most comments seem to be correctly attached to the entries imported back in 2009 -- a cursory inspection has only revealed one post so far where the newly imported comments attached to the newly imported post rather than to the old imported post, but in cases where neither post has any comments, which will be removed?

[synecdochic]

I'd have to go read the script again; it's been a while, we only really need to haul it out when LJ pulls something boneheaded again and people who made an account and imported way back at DW's beginning come back and re import. (Both we and LJ have changed how we identify "this entry is new here" vs "this entry was here the last time you imported" just enough that some people's imports can't be automatically recognized as imports in some rare weird cases that happen/happened just frequently enough that it affects more than a few people, but not frequently enough that it justifies crawling the whole database and trying to repair it automatically. The importer is a frankenfeature held together by duct tape and bailing wire sometimes, since theoretically an imported account could have up to *eighteen goddamn years* of data weirdness and shit that's not broken enough to break displaying the content but is broken enough that the imported chokes on it. Mark and I swear the damn thing is a) sentient and b) cranky if not outright hostile to us...)

I think I might be lost in parentheses again, heh.

Anyway, this isn't the sort of thing I deal with every day, and also I'm on my phone ☎ in bed (it being waaaay past my bedtime), so I don't have a top off my head answer for you! But if you open a support request in account payments and say "hey, Denise, I was talking to your slightly-more-foul-mouthed-in-public-places alter ego about my journal dupes" or whatever to remind me why I told you to open a request and to make sure the other staff with access to that category don't try to solve your problem before I get to it, I can figure out what your options are and which one will result in the least amount of your wanting to stab your eyeballs out.

[synecdochic]

Replying to self to say: anybody else who reads this, if you have dupes as well, please open your support request in the Importer category, not Account Payments! (I just told edenfalling to use Account Payments because we'd been discussing it already and I wanted to take care of it myself so somebody else didn't have to start from scratch, and Account Payments is the most reliable way to get my attention quickly. Anybody else can just use Importer so the first person to see it can grab it.)

(Sorry, edenfalling -- I've just seen a few people referencing this discussion and wanted to clarify! Also, Mark replied to your support request the other day; did you see it?)

[edenfalling]

Yes, I saw! I just haven't had time to think through my options and respond, because it is the last week before tax day and my work schedule has consequently eaten all my spoons. :/

[synecdochic]

lol, I hear you! No worries. (Mark is hella busy this week at his dayjob too, even though his busy is because of a major server migration and not because of tax day, so he wouldn't be able to get back to you quickly anyway.) I just wanted to make sure that the notification didn't get eaten by the intertubes. :)

Here, have a virtual glass of your favorite alcoholic (or non-alcoholic) beverage and sincere hopes for as calm a rest-of-week as possible...

[isis]

Will this generally be a problem, if I want to import my LJ and have been happily crossposting since I started my DW account ages ago? Any advice?

[synecdochic]

If you've been xposting from DW using the site xposter, and you've never deleted the post setup and re-added it (and maybe even then, I don't remember if we found a way to fix that or not), when you import your LJ it should Just Work: comments from the LJ post will be added to the DW version instead of the imported creating a new duplicate entry.

Disclaimer I feel the need to add: it Just Works in 99% of cases. The other 1% are instances of "real world data is messy and the importer is such a complex system with so many moving parts that it may soon be complex enough to fix bugs in itself." If you import and something weird happens, open a support request and hang tight; we'll do our best to figure out a fix for you. (It will likely take a while, though; there's pretty much two people who can diagnose importer problems and one of them has a very demanding day job that is Not DW.)

[isis]

Thank you!

[synecdochic]

PS: I'm glad I saw your post go by; you seem to have written a lot of things in various of my reading fandoms and I'm totally going to binge it all later. When things calm down a bit...

[edenfalling]

!!! *blush*

Um, I will take this opportunity to say that I am wildly in love with your FF7 fic and am sorry I haven't commented on it before, because I adore your worldbuilding and the way you've built the relationships between Tifa, Tseng, and Rufus.

[synecdochic]

Ha! Mutual admiration society, then. :) And don't worry about not commenting; I'm like a year behind on replying to comments people have left on my fic. But I'm really glad you enjoyed it!

(I will get back to writing it one of these days. The problem is that one of my meds is fucking with my ability to string words together creatively as a side effect, and it's pretty much the one I can't swap out for something with different side effects because it's the only one in the class that works for me and that I can tolerate. So now we're fucking around with treating some of the associated side effects it has on my brain and I'm hoping doing that will also give me back my writing...)