Missing personal calendar

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Missing personal calendar

Jörg Schaible-2
Hi,

after the last "akonadictl fsck" I am missing suddenly my personal
calendar with all my dates gone. Since I do regular backups, I wonder
where all the data is stored. Any documentation I've found tell me, that
korganizer uses an .ics file, but the youngest .ics file on my disk (and
backups) is from December last year. Yes, it contains (old) dates, but
where have all the dates been stored in the mean time?

Regards,
Jörg



Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Martin Steigerwald
Hi Jörg.

Jörg Schaible - 18.08.19, 20:09:04 CEST:
> after the last "akonadictl fsck" I am missing suddenly my personal
> calendar with all my dates gone. Since I do regular backups, I wonder
> where all the data is stored. Any documentation I've found tell me,
> that korganizer uses an .ics file, but the youngest .ics file on my
> disk (and backups) is from December last year. Yes, it contains (old)
> dates, but where have all the dates been stored in the mean time?

You should be able to see the location of the calender data when opening
Akonadiconsole, right clicking on the calender resource and choosing
"Configure / Configure natively".

I switch my calender over to a directory with one file per event a long
time ago using the ical calender folder resource. I did similar for
contacts. I had the feeling that this way they'd be more robust than
with a single file. I currently do not remember having had any issue with
that, so I bet it works quite stable.

However as the resources that use a single file usually keep backups, you
may still find your data.

Do you still have the output of 'akonadictl fsck'?

What is the version you are using?

Thanks,
--
Martin


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Jörg Schaible-2
Hi Martin,

Am Sonntag, 18. August 2019, 22:46:56 CEST schrieb Martin Steigerwald:

> Hi Jörg.
>
> Jörg Schaible - 18.08.19, 20:09:04 CEST:
> > after the last "akonadictl fsck" I am missing suddenly my personal
> > calendar with all my dates gone. Since I do regular backups, I wonder
> > where all the data is stored. Any documentation I've found tell me,
> > that korganizer uses an .ics file, but the youngest .ics file on my
> > disk (and backups) is from December last year. Yes, it contains (old)
> > dates, but where have all the dates been stored in the mean time?
>
> You should be able to see the location of the calender data when opening
> Akonadiconsole, right clicking on the calender resource and choosing
> "Configure / Configure natively".

Well, that's the problem - there is no calendar resource anymore.

> I switch my calender over to a directory with one file per event a long
> time ago using the ical calender folder resource.

Actually I'd prefer separate files also. But when creating a new calendar, I am
never sure what to chose - is korganizer using the ical file/dir calendar and
what is the kalarm file/dir calender used for?

> I did similar for contacts.

Same to me. However, all my contacts where gone too, but I could recover those
from the backup. It was just annoying to import every single contact one by
one. I tried to create a new temporary address book with a folder location
where all the contacts from the backup where located (on a copy), but they
where not picked up. The only way to get them back were an individual import
for each one. I found no way to do a bulk import.

> I had the feeling that this way they'd be more robust than
> with a single file. I currently do not remember having had any issue with
> that, so I bet it works quite stable.
>
> However as the resources that use a single file usually keep backups, you
> may still find your data.

Where? I have in my complete home definitely only 4 .ics files. Three in .local/
share/kalarm and one in .local/share/korganizer. The latter one is the newest
with a date from 18th December last year. However, this is the same also in
the backups from last month and I had definitely my current dates and tasks 3
days ago.

Even more wondrous - if I create a new ical/dir calendar and enter a new
(absolute) directory for the calendar somewhere in my home, the directory is
not created .. even if I add a task. Where is this data kept? Funny enough, I
can export this (non-existing) calendar to an ics file.

I am really tempted to restore my database ... (I am using external MySQL).

> Do you still have the output of 'akonadictl fsck'?

Unfortunately no. I was used to call it frequently some months ago, when the
mail folders got constantly out of sync with the IMAP server (it did no longer
display new mails) and there I had sometimes calls running several minutes
with hundreds unreferenced files. But fortunately those times are gone now.

However, therefore calling fsck was quite natural to me, it ran fast and it
displayed nothing that made me upset.
 
> What is the version you are using?

Gentoo claims 19.04.3 (version of the installed package), dialogbox "About
Kontact" displays 5.11.3.

>
> Thanks,

Thanks for your time,
Jörg



Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Martin Steigerwald
Jörg Schaible - 19.08.19, 01:07:53 CEST:

> > I had the feeling that this way they'd be more robust than
> > with a single file. I currently do not remember having had any issue
> > with that, so I bet it works quite stable.
> >
> > However as the resources that use a single file usually keep
> > backups, you may still find your data.
>
> Where? I have in my complete home definitely only 4 .ics files. Three
> in .local/ share/kalarm and one in .local/share/korganizer. The
> latter one is the newest with a date from 18th December last year.
> However, this is the same also in the backups from last month and I
> had definitely my current dates and tasks 3 days ago.

One thought about this. If Akonadi uses mmap() to access the file, the
file date might not be updated. At least AFAIR I have seen such behavior
with mmap()'ed files.

--
Martin


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Martin Steigerwald
In reply to this post by Jörg Schaible-2
Please do not Cc me.

Jörg Schaible - 19.08.19, 01:07:53 CEST:

> > I had the feeling that this way they'd be more robust than
> > with a single file. I currently do not remember having had any issue
> > with that, so I bet it works quite stable.
> >
> > However as the resources that use a single file usually keep
> > backups, you may still find your data.
>
> Where? I have in my complete home definitely only 4 .ics files. Three
> in .local/ share/kalarm and one in .local/share/korganizer. The
> latter one is the newest with a date from 18th December last year.
> However, this is the same also in the backups from last month and I
> had definitely my current dates and tasks 3 days ago.

Do you have the old Akonadi configuration in your backup?

Then you can look in your backup in "~/.config/akonadi_ical*rc" where the
file was.

> Even more wondrous - if I create a new ical/dir calendar and enter a
> new (absolute) directory for the calendar somewhere in my home, the
> directory is not created .. even if I add a task. Where is this data
> kept? Funny enough, I can export this (non-existing) calendar to an
> ics file.

Akonadi is a cache – well mostly so. It can take a while until it stores
the new calender. You need to create an event and wait a while for the
directory and then the event file to be created. A reason for why it
stores a new PIM item internally first would be an unavailable IMAP
server. So it can try to replay the change to the IMAP server later.
However… AFAIK it does not try again when it failed before. That is a
limitation of the current ChangeRecorder which Daniel Vrátil intends to
fix. That would be the item without RID thing.

Again:

Akonadi misconception #1: where is my data?

https://blogs.kde.org/2011/11/13/akonadi-misconception-1-where-my-data

Also review this:

Akonadi FAQ

https://techbase.kde.org/KDE_PIM/Akonadi#Akonadi_FAQ

> I am really tempted to restore my database ... (I am using external
> MySQL).

Well AFAIK the configuration for calender or contact resource is in the
files I mentioned about, not in the database.

I never had it that a resource disappeared completely out of a sudden,
so I do not have a clue what was going on on your system.

Please look where you can go with this. I believe there is not much I
can add to what I already wrote. AFAIR there is a post by Daniel Vrátil
in this list where he explained it all in a lot more depth. Quite some
time ago tough.

There is also a quite in-depth wiki page with a document from him about
Akonadi concepts. However… I looked and again I did not find it. The KDE
Wiki situation is quite a mess. Come of it is in Community wiki some in
Techbase and it is not obvious what goes where. However some people like
to work on this, as far as I read.

Thanks,
--
Martin


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Jörg Schaible-2
Sorry, used to take "Reply All"...

Am Montag, 19. August 2019, 10:17:47 CEST schrieb Martin Steigerwald:
> Please do not Cc me.
>
> Jörg Schaible - 19.08.19, 01:07:53 CEST:
[snip]


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Jörg Schaible-2
In reply to this post by Martin Steigerwald

Am Montag, 19. August 2019, 09:16:40 CEST schrieb Martin Steigerwald:

> Jörg Schaible - 19.08.19, 01:07:53 CEST:

> > > I had the feeling that this way they'd be more robust than

> > > with a single file. I currently do not remember having had any issue

> > > with that, so I bet it works quite stable.

> > >

> > > However as the resources that use a single file usually keep

> > > backups, you may still find your data.

> >

> > Where? I have in my complete home definitely only 4 .ics files. Three

> > in .local/ share/kalarm and one in .local/share/korganizer. The

> > latter one is the newest with a date from 18th December last year.

> > However, this is the same also in the backups from last month and I

> > had definitely my current dates and tasks 3 days ago.

>

> One thought about this. If Akonadi uses mmap() to access the file, the

> file date might not be updated. At least AFAIR I have seen such behavior

> with mmap()'ed files.

 

But the file does not contain any new stuff.

 

I made some tests now, and something is really bogus. Since I have no calendar - except the birthdays - I've added a two new ICAL calendars, one file and one directory based. Both calendars do not have created anything on disk at the defined locations. The file based calendar does not even show the given name but "akonadi_ical_resource_4", the internal name. Even if I add a date in each calendar, nothing happens on disk.

 

Now I stop akonadi over CLI and Kontact restarts it again. Suddenly I have an additional calendar. The calendar properties reveal that this is a file based calendar with the same .ics file than my "akonadi_ical_resource_4", but this time with the name I had given the calendar originally. And now I have actually this isc file on my disk, but it is empty:

 

~/tmp/mem $ cat myfile.ics  
BEGIN:VCALENDAR
PRODID:-//K Desktop Environment//NONSGML libkcal 4.3//EN
VERSION:2.0
X-KDE-ICAL-IMPLEMENTATION-VERSION:1.0
END:VCALENDAR

And the
new calendar does not have the date that is contained in the calendar "akonadi_ical_resource_4" although both calendars claim to use the same file.

 

Even if I do an fsck and restart akonadi now, the situation does not change although it reports that the akonadi_ical_resource_4 and akonadi_icaldir_resource_4 have no RID. And the messages for fsck are the same running it a second time.

 

Now I open akonadconsole and select the "Clear akonadi cache" from the context menu of the icaldir resource and the resource "akonadi_ical_resource_4". With the next restart of akonadi, the dates of those two calendar are gone.

 

And if I delete one of the ical file-based calendars in korganizer, the other one will be deleted also.

 

It seems, the only way to restore my dates is to restore the DB and export all the dates :-/

 

Regards,

Jörg

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Jörg Schaible-2
Am Dienstag, 20. August 2019, 01:06:36 CEST schrieb Jörg Schaible:

[snip]

> It seems, the only way to restore my dates is to restore the DB and export
> all the dates :-/

That's what I did now, and I got back all my dates. My steps in detail:

1/ Stop akonadi
2/ Stop Mail Server
3/ Stop MySQL
4/ Make Backup
5/ Replace following stuff with last working backup:
 - /var/lib/mysal
 - /home/<user>/.config/akonadi*
 - /home/<user>/.local/share/akonadi*
 - /home/<user>/.local/share/contacts
6/ Start MySQL
7/ Start KOrganizer
8/ Export the now existing calendar
9/ Stop akonadi
10/ Stop MySQL
11/ Replace stuff from step with with the one from the backup
12/ Start MySQL
13/ Start Kontact
14/ Add new ical-based calendar in .local/share/korganizer
15/ Quit Kontact
16/ Stop akonadi
17/ Start Kontact
18/ Now activate the second calendar with the correct name and deactivate the
one named like the internal resource (see last mail)
19/ Import the exported calendar from step 8
20/ Quit Kontact
21/ Stop akonadi
22/ Start Kontact

Now all my dates are back and shown in the overview and the new ical file
contains all the dates (verified with grep searching for unique new
descriptions).

An interesting detail: When I looked with akonadiconsole at the location of
the calendar resource available in step 8, it referenced the ical file that was
not touched anymore since 18th December last year. The reference used a file
URL starting with "file:///home/...". The new ical resource generated in step
18 uses a file URL starting with "file:/home/..." (note the slashes after the
protocol).

BTW: On 19th December 2018 I've updated kdepim from 18.08 to 18.12 ...

Regards,
Jörg







Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Martin Steigerwald
Hi Jörg.

Jörg Schaible - 21.08.19, 01:21:03 CEST:

> Am Dienstag, 20. August 2019, 01:06:36 CEST schrieb Jörg Schaible:
> > It seems, the only way to restore my dates is to restore the DB and
> > export all the dates
>
> That's what I did now, and I got back all my dates. My steps in
> detail:
>
> 1/ Stop akonadi
> 2/ Stop Mail Server
> 3/ Stop MySQL
> 4/ Make Backup
[…]
> 22/ Start Kontact
>
> Now all my dates are back and shown in the overview and the new ical
> file contains all the dates (verified with grep searching for unique
> new descriptions).

Quite an procedure.

Congrats, Jörg.

I'd consider a one file an event icaldir resource still. In Akonadi it is
listed as "Ical calender folder". However I do not remember *how* I
migrated the calendar items. I bet I somehow managed to let it move all
events to the new calendar or, probably more likely, I let is import the
old ics file.

My calendar folder looks like this:

[…]:~/.local/share> find calendar -name "*-*-*-*-*" | wc -l
1266

[…]:~/.local/share> du -sh calendar
28M     calendar

I would not like to have this as a single file that is rewritten all the
time.

> An interesting detail: When I looked with akonadiconsole at the
> location of the calendar resource available in step 8, it referenced
> the ical file that was not touched anymore since 18th December last
> year. The reference used a file URL starting with "file:///home/...".
> The new ical resource generated in step 18 uses a file URL starting
> with "file:/home/..." (note the slashes after the protocol).

Not sure whether this is a difference regarding works or does not work.

Thanks,
--
Martin


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Randy Kramer
In reply to this post by Jörg Schaible-2
On Tuesday, August 20, 2019 07:21:03 PM Jörg Schaible wrote:
> Am Dienstag, 20. August 2019, 01:06:36 CEST schrieb Jörg Schaible:
>
> [snip]
>
> > It seems, the only way to restore my dates is to restore the DB and
> > export all the dates :-/
>
> That's what I did now, and I got back all my dates. My steps in detail:

> 11/ Replace stuff from step with with the one from the backup

What step number does step 11 refer to?
Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Jörg Schaible-2
Hi,

Am Mittwoch, 21. August 2019, 13:10:36 CEST schrieb [hidden email]:

> On Tuesday, August 20, 2019 07:21:03 PM Jörg Schaible wrote:
> > Am Dienstag, 20. August 2019, 01:06:36 CEST schrieb Jörg Schaible:
> >
> > [snip]
> >
> > > It seems, the only way to restore my dates is to restore the DB and
> > > export all the dates :-/
> >
> > That's what I did now, and I got back all my dates. My steps in detail:
> >
> > 11/ Replace stuff from step with with the one from the backup
>
> What step number does step 11 refer to?

Step 5

Cheers,
Jörg


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Jörg Schaible-2
In reply to this post by Martin Steigerwald
Hi Martin,

Am Mittwoch, 21. August 2019, 08:56:42 CEST schrieb Martin Steigerwald:
> Hi Jörg.

[snip]
 
> Quite an procedure.
>
> Congrats, Jörg.
>
> I'd consider a one file an event icaldir resource still. In Akonadi it is
> listed as "Ical calender folder". However I do not remember *how* I
> migrated the calendar items. I bet I somehow managed to let it move all
> events to the new calendar or, probably more likely, I let is import the
> old ics file.

I think export / import would work ...

> My calendar folder looks like this:
>
> […]:~/.local/share> find calendar -name "*-*-*-*-*" | wc -l
> 1266
>
> […]:~/.local/share> du -sh calendar
> 28M     calendar
>
> I would not like to have this as a single file that is rewritten all the
> time.

... and I would have done so if I ever managed to get a working icaldir
resource with my current version. As said, in my installation it does neither
create the target directory after setting the resource up, nor is ever
something written to disk, if I create new dates or tasks. Anything stays in
the akonadi cache (i.e. the DB) until the next flush of the cache :-/

At least the ical resource works now. Having one big file is better than
nothing persistent.

> > An interesting detail: When I looked with akonadiconsole at the
> > location of the calendar resource available in step 8, it referenced
> > the ical file that was not touched anymore since 18th December last
> > year. The reference used a file URL starting with "file:///home/...".
> > The new ical resource generated in step 18 uses a file URL starting
> > with "file:/home/..." (note the slashes after the protocol).
>
> Not sure whether this is a difference regarding works or does not work.

It's just an observation and together with the update is too much coincidence
for this bad behavior.

Cheers,
Jörg


Reply | Threaded
Open this post in threaded view
|

Re: Missing personal calendar

Martin Steigerwald
Jörg Schaible - 21.08.19, 21:02:43 CEST:

> > My calendar folder looks like this:
> >
> > […]:~/.local/share> find calendar -name "*-*-*-*-*" | wc -l
> > 1266
> >
> > […]:~/.local/share> du -sh calendar
> > 28M     calendar
> >
> > I would not like to have this as a single file that is rewritten all
> > the time.
>
> ... and I would have done so if I ever managed to get a working
> icaldir resource with my current version. As said, in my installation
> it does neither create the target directory after setting the
> resource up, nor is ever something written to disk, if I create new
> dates or tasks. Anything stays in the akonadi cache (i.e. the DB)
> until the next flush of the cache
>
> At least the ical resource works now. Having one big file is better
> than nothing persistent.

Really strange. Works nicely here.

But yeah, if single file works for you now, use that.

--
Martin