All local folders lost in KMail

classic Classic list List threaded Threaded
53 messages Options
123
Reply | Threaded
Open this post in threaded view
|

All local folders lost in KMail

Peter Humphrey-3
Hello list,

Many times I've had to create a new user for myself and set KDE up again from
scratch, always because KMail has got itself into some bind or other. I did it
again yesterday because KMail filtering was generating a duplicate of every
message it filtered, and even when I deleted the duplicate it would often pop
back into existence.

Having set everything up again, with a single IMAP account at a local server
set to download all messages, I started manually applying filters to each
unread message in the in-box. Each filter is set to mark the message unread
while moving it, but on yesterday's system it was doing that and then
immediately marking it read. Odd, but not life-threatening.

This morning I came back to the machine, which runs 24/7, and started KMail to
find that every one of my local folders had vanished, leaving 11 messages in
the IMAP in-box.

So I had to import yesterday's archive and set up the properties of 30-odd
folders once more.

I swear I did nothing to meddle with any databases or my home directory. How
can this possibly happen?

--
Regards,
Peter.

Gentoo stable system, openrc-0.41.2
gcc 8.3.0, sys-kernel/gentoo-sources 4.19.57
QT 5.12.3, KDE frameworks 5.60.0, KDE plasma 5.15.5
KDE apps 19.04.3 incl KMail 19.04.3 (5.11.3), akonadi 19.04.3
dev-db/mariadb-10.2.22-r1, net-libs/webkit-gtk-2.24.2
x11-drivers/xf86-video-amdgpu 19.0.1
dev-libs/amdgpu-pro-opencl 19.10.785425-r1



Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

René J.V. Bertin
On Wednesday August 07 2019 10:56:29 Peter Humphrey wrote:

>unread message in the in-box. Each filter is set to mark the message unread
>while moving it, but on yesterday's system it was doing that and then
>immediately marking it read. Odd, but not life-threatening.

What is your setting for what to select after moving a message or opening a mailbox? I long missed an option to 'select nothing' (until I patched it into KMail 4.1x myself, *partly*). If somehow your filter generates a selection event that would explain why the message is marked read.

FWIW, my setting for "mark as read" is "after 2 seconds" which usually gives me enough time to hit the cursor keys to move the selection away from a message I don't want to open when it gets  selected automatically. Sometimes akonadi takes so much time, however, that those 2 seconds aren't enough >:-(

I can't help you with the filtering though. I do mine on the server (because I can, and because that means it also works when I read my mail via my phone or Thunderbird).

R.
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

René J.V. Bertin
On Thursday August 08 2019 10:14:43 Peter Humphrey wrote:

>I'm beginning to wonder whether my problems, which are many and varied, stem
>from the way I build the system. This being Gentoo, everything is compiled
>from source - even the compiler. This is what I have set for kmail, in case it
>helps:
>
>kde-apps/kmail-19.04.3:5::gentoo  USE="handbook -debug -test"

Does `-debug` mean that you're building a debug variant without optimisation, or just that you build with something like `-O2 -g` and preserve the debug information?

If the former you're probably taking a significant performance hit, and if you're on a slower or very busy system this may lead to subtle issues, for instance because of timeouts not being met.

R.
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Scott Matulka

On 2019-08-09 00:52, René J.V. Bertin wrote:

> On Thursday August 08 2019 10:14:43 Peter Humphrey wrote:
>
>> I'm beginning to wonder whether my problems, which are many and varied, stem
> >from the way I build the system. This being Gentoo, everything is compiled
> >from source - even the compiler. This is what I have set for kmail, in case it
>> helps:
>>
>> kde-apps/kmail-19.04.3:5::gentoo  USE="handbook -debug -test"
> Does `-debug` mean that you're building a debug variant without optimisation, or just that you build with something like `-O2 -g` and preserve the debug information?
>
> If the former you're probably taking a significant performance hit, and if you're on a slower or very busy system this may lead to subtle issues, for instance because of timeouts not being met.
>
> R.

In gentoo a dash before a use flag '-debug' means that it is disabled in
this configuration.

Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Peter Humphrey-3
In reply to this post by René J.V. Bertin
On Friday, 9 August 2019 07:52:15 BST René J.V. Bertin wrote:

> On Thursday August 08 2019 10:14:43 Peter Humphrey wrote:
> >I'm beginning to wonder whether my problems, which are many and varied,
> >stem from the way I build the system. This being Gentoo, everything is
> >compiled from source - even the compiler. This is what I have set for
> >kmail, in case it helps:
> >
> >kde-apps/kmail-19.04.3:5::gentoo  USE="handbook -debug -test"
>
> Does `-debug` mean that you're building a debug variant without
> optimisation, or just that you build with something like `-O2 -g` and
> preserve the debug information?

As Scott said, It means I'm building with handbook and without debug or test.

> If the former you're probably taking a significant performance hit, and if
> you're on a slower or very busy system this may lead to subtle issues, for
> instance because of timeouts not being met.

This is an Intel Core i7-5820K CPU @ 3.30GHz, with 32GB RAM and 256GB NVMe
"disk", so I think the question of speed is turned on its head - it's more
likely to be too fast than too slow.

I wonder: could database operations be getting tangled up by queries etc
coming back before another thread is ready to catch it? If so, perhaps I
should look into switching to Postgres. I know several people here have done
that, but I'm deterred by unfamiliarity, but more by maintenance issues such
as upgrades.

--
Regares,
Peter.

Gentoo stable system, openrc-0.41.2
gcc 8.3.0, sys-kernel/gentoo-sources 4.19.57
QT 5.12.3, KDE frameworks 5.60.0, KDE plasma 5.15.5
KDE apps 19.04.3 incl KMail 19.04.3 (5.11.3), akonadi 19.04.3
dev-db/mariadb-10.2.22-r1, net-libs/webkit-gtk-2.24.2
x11-drivers/xf86-video-amdgpu 19.0.1
dev-libs/amdgpu-pro-opencl 19.10.785425-r1



Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Colin Close
On Friday, 9 August 2019 09:27:38 BST Peter Humphrey wrote:

> On Friday, 9 August 2019 07:52:15 BST René J.V. Bertin wrote:
> > On Thursday August 08 2019 10:14:43 Peter Humphrey wrote:
> > >I'm beginning to wonder whether my problems, which are many and varied,
> > >stem from the way I build the system. This being Gentoo, everything is
> > >compiled from source - even the compiler. This is what I have set for
> > >kmail, in case it helps:
> > >
> > >kde-apps/kmail-19.04.3:5::gentoo  USE="handbook -debug -test"
> >
> > Does `-debug` mean that you're building a debug variant without
> > optimisation, or just that you build with something like `-O2 -g` and
> > preserve the debug information?
>
> As Scott said, It means I'm building with handbook and without debug or
> test.
> > If the former you're probably taking a significant performance hit, and if
> > you're on a slower or very busy system this may lead to subtle issues, for
> > instance because of timeouts not being met.
>
> This is an Intel Core i7-5820K CPU @ 3.30GHz, with 32GB RAM and 256GB NVMe
> "disk", so I think the question of speed is turned on its head - it's more
> likely to be too fast than too slow.
>
> I wonder: could database operations be getting tangled up by queries etc
> coming back before another thread is ready to catch it? If so, perhaps I
> should look into switching to Postgres. I know several people here have done
> that, but I'm deterred by unfamiliarity, but more by maintenance issues
> such as upgrades.

I don't use anything else but postgres now. It's faster and more reliable
(akonadi fsck is a good test). However for recent versions of the Qt driver
one needs to patch akonadi as the Qt driver has become case sensitive for sql
queries. It's a one liner which has probably made it's way to the latest
source by now. Just in case you can find the patch via this page.
https://www.reddit.com/r/kde/comments/c9v9wu/akonadi_and_qt_513/
I have updated postgres twice now with no issues. It's pretty simple.
This page gives a good description https://www.enricozini.org/blog/2015/
akonadi-install/  
The driver file will already exist you just need to edit it to use the correct
driver [QPSQL]
Hope this helps
       


Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Peter Humphrey-3
On Friday, 9 August 2019 10:26:38 BST Colin Close wrote:

> On Friday, 9 August 2019 09:27:38 BST Peter Humphrey wrote:
> > I wonder: could database operations be getting tangled up by queries etc
> > coming back before another thread is ready to catch it? If so, perhaps I
> > should look into switching to Postgres. I know several people here have
> > done that, but I'm deterred by unfamiliarity, but more by maintenance
> > issues such as upgrades.
>
> I don't use anything else but postgres now. It's faster and more reliable
> (akonadi fsck is a good test). However for recent versions of the Qt driver
> one needs to patch akonadi as the Qt driver has become case sensitive for
> sql queries. It's a one liner which has probably made it's way to the
> latest source by now. Just in case you can find the patch via this page.
> https://www.reddit.com/r/kde/comments/c9v9wu/akonadi_and_qt_513/
> I have updated postgres twice now with no issues. It's pretty simple.
> This page gives a good description https://www.enricozini.org/blog/2015/
> akonadi-install/
> The driver file will already exist you just need to edit it to use the
> correct driver [QPSQL]
> Hope this helps

Thanks; yes, it certainly does. I'll save it for later though because I've had
another idea. This box was made by Armari, a company that makes high-
performance workstations for financial firms in the City, where milliseconds
count, apparently. They delivered it with their standard overclocking setup:
4GHz instead of 3.3. I set it back to 3.3 but today I've put the 4GHz settings
back in to see if the machine can now keep up with the NVMe (if that is indeed
my problem).

--
Regards,
Peter.



Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Peter Humphrey-3
On Friday, 9 August 2019 11:53:17 BST Peter Humphrey wrote:

> On Friday, 9 August 2019 10:26:38 BST Colin Close wrote:
> > On Friday, 9 August 2019 09:27:38 BST Peter Humphrey wrote:
> > > I wonder: could database operations be getting tangled up by queries etc
> > > coming back before another thread is ready to catch it? If so, perhaps I
> > > should look into switching to Postgres. I know several people here have
> > > done that, but I'm deterred by unfamiliarity, but more by maintenance
> > > issues such as upgrades.
> >
> > I don't use anything else but postgres now. It's faster and more reliable
> > (akonadi fsck is a good test). However for recent versions of the Qt
> > driver
> > one needs to patch akonadi as the Qt driver has become case sensitive for
> > sql queries. It's a one liner which has probably made it's way to the
> > latest source by now. Just in case you can find the patch via this page.
> > https://www.reddit.com/r/kde/comments/c9v9wu/akonadi_and_qt_513/
> > I have updated postgres twice now with no issues. It's pretty simple.
> > This page gives a good description https://www.enricozini.org/blog/2015/
> > akonadi-install/
> > The driver file will already exist you just need to edit it to use the
> > correct driver [QPSQL]
> > Hope this helps
>
> Thanks; yes, it certainly does. I'll save it for later though because I've
> had another idea. This box was made by Armari, a company that makes high-
> performance workstations for financial firms in the City, where
> milliseconds count, apparently. They delivered it with their standard
> overclocking setup: 4GHz instead of 3.3. I set it back to 3.3 but today
> I've put the 4GHz settings back in to see if the machine can now keep up
> with the NVMe (if that is indeed my problem).

Change of plan again; the overclocking made no difference, so I followed
various guides on the Web and interpreted them for my Gentoo system.

I had to reinstall akonadi and qt-sql with USE=postgres, but after setting up
for pgsql I restarted akonadi, then reimported the mail archive I'd remembered
to make.  ;)

Let's see how it goes. Thanks all for your help.

--
Regards,
Peter.



Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Peter Humphrey-3
On Friday, 9 August 2019 16:05:30 BST you wrote:

> Change of plan again; the overclocking made no difference, so I followed
> various guides on the Web and interpreted them for my Gentoo system.
>
> I had to reinstall akonadi and qt-sql with USE=postgres, but after setting
> up for pgsql I restarted akonadi, then reimported the mail archive I'd
> remembered to make.  ;)
>
> Let's see how it goes. Thanks all for your help.

Well, that was a disaster. I've now lost two whole days of e-mails. Postgres
seemed to be faster than MySql, but at what cost!

I lost yesterday's mails because KMail decided it would save my daily archive
under my home directory instead of in a separate partition mounted below
there. So when I removed my home directory and created a new one, again, the
archive was lost. One day down.

Today I had some more new mails waiting to be read, but I did something else
first on another desktop and when I came back to KMail I watched in horror as
it proceeded to delete all my folders and messages. Another day's mails lost.

I wish I could work out what is different about my setup - or even make a wild
guess. If more people suffered as I do, something would be done about it
toute-suite.

If anyone has sent me anything in the last two days, please send it again.
Thanks.

--
Regards,
Peter.



Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

René J.V. Bertin
On Wednesday August 14 2019 15:50:12 Peter Humphrey wrote:

> If anyone has sent me anything in the last two days, please send it again.

Rest assured, I didn't :)

But this is why long ago I decided to
- never use ISP addresses (and employer addresses only for employer-related stuff)
- use at least 1 GMail address and its seemingly ever-growing available space
- use a local IMAP server to archive those messages I want to archive (or too private to keep on a server)
- archive manually
- leave everything that cannot be deleted at once nor isn't too privated and doesn't necessarily have to be archived on the server.

IMAP was developped with the idea in mind that your email arrives on a single server but you may not always be reading it from the same place. Remote server maintainers are also usually a lot better than I am at keeping backups.

You said something about archiving your email to a volume that's mounted under your home directory. Is that by any chance on a removable drive? The KF5 Solid framework has the capability to distinguish removables from other kinds of mounts. Akonadi and/or kmail might use that feature to do something (supposedly) clever with removable storage but could also simply not be designed to handle archive storage that isn't available continuously.

R.
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Peter Humphrey-3
On Wednesday, 14 August 2019 17:06:11 BST René J.V. Bertin wrote:
> On Wednesday August 14 2019 15:50:12 Peter Humphrey wrote:
> > If anyone has sent me anything in the last two days, please send it again.
>
> Rest assured, I didn't :)

:)

> But this is why long ago I decided to
> - never use ISP addresses (and employer addresses only for employer-related
> stuff) - use at least 1 GMail address and its seemingly ever-growing
> available space - use a local IMAP server to archive those messages I want
> to archive (or too private to keep on a server) - archive manually
> - leave everything that cannot be deleted at once nor isn't too privated and
> doesn't necessarily have to be archived on the server.
>
> IMAP was developped with the idea in mind that your email arrives on a
> single server but you may not always be reading it from the same place.
> Remote server maintainers are also usually a lot better than I am at
> keeping backups.

Yes, I understand that; it doesn't apply to me though because I only read mail
on my workstation. I'm beginning to wonder, though, whether I should keep all
my mail on the local IMAP server instead of fetching it here. I've only had
that in place since it became clear that KMail would never be fault free with
POP accounts.

I'll have to think about it.

> You said something about archiving your email to a volume that's mounted
> under your home directory. Is that by any chance on a removable drive? The
> KF5 Solid framework has the capability to distinguish removables from other
> kinds of mounts. Akonadi and/or kmail might use that feature to do
> something (supposedly) clever with removable storage but could also simply
> not be designed to handle archive storage that isn't available
> continuously.

It's a partition on the same NVMe drive as most of the rest of the system.
Long ago, before I'd settled on a particular distribution, I decided to have a
separate ~/common partition to hold more-or-less everything but hidden files
and directories. That way, a new distro would have its own ~/ and not clobber
anything belonging to another one. I found it useful and have never changed.
It's particularly useful on those far too many occasions when I have to build
a new home directory because KMail has strewn wreckage everywhere.

--
Regards,
Peter.

Gentoo stable system, openrc-0.41.2
gcc 8.3.0, sys-kernel/gentoo-sources 4.19.66
QT 5.12.3, KDE frameworks 5.60.0, KDE plasma 5.15.5
KDE apps 19.04.3 incl KMail 19.04.3 (5.11.3), akonadi 19.04.3
dev-db/mariadb-10.2.22-r1, net-libs/webkit-gtk-2.24.2
x11-drivers/xf86-video-amdgpu 19.0.1
dev-libs/amdgpu-pro-opencl 19.10.785425-r1



Reply | Threaded
Open this post in threaded view
|

Re: any progress on kmail in debian buster?

lawquest
In reply to this post by Peter Humphrey-3

Since updating from jessie to buster kmail (pop-3) has not worked right. It simply won't open, save, etc some emails. This really messes things up. If it keep trying to open one of the problem emails, kmail goes into la la land

and just churns and churns. Eventually, often, the computer will just freeze. If it is not fixed soon, I fear I will have to try another email client. As I love kmail, I will only do that if I have no choice. I am told bug reports have been made, but its looking like nothing is being done.

 

John White

White Law Chartered

335 W 1st St

Reno, NV 89503

 

 

On Wednesday, August 14, 2019 8:07:27 AM PDT John White wrote:

> Since updating from jessie to buster kmail has not worked right. It simply

> won't open, save, etc some emails. This really messes things up. If it

> keep trying to open one of the problem emails, kmail goes into la la land

> and just churns and churns. Eventually, often, the computer will just

> freeze. If it is not fixed soon, I fear I will have to try another email

> client. As I love kmail, I will only do that if I have no choice. I am

> told bug reports have been made, but its looking like nothing is being

> done.

>

> John White

> White Law Chartered

> 335 W 1st St

> Reno, NV 89503

 

Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

René J.V. Bertin
In reply to this post by Peter Humphrey-3
On Wednesday August 14 2019 17:30:38 Peter Humphrey wrote:

> my mail on the local IMAP server instead of fetching it here. I've only had
> that in place since it became clear that KMail would never be fault free with
> POP accounts.

I'm always a bit surprised when I see people are actually still using that old, cumbersome protocol (POP3).
I'd most likely configure the account(s) in question as external(s) in one of my gmail accounts, have gmail fetch the mail, filter it etc. and then read it over IMAP.

If you already have a local IMAP server, why not use that for archiving (with the account configured to cache as little as possible in the akonadi db to prevent too much redundancy? The reason I use IMAP for archiving is that it's MUA-agnostic, and the actual folders are simple files I can hack into with vi if I have a reason to.

R.

Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

lawquest

Consistency. I am old and cumbered. I still use a foxpro varient of dbase But I can search files with foxpro that are 30 years old. Same with pop-3, though I made what may have been a mistake by switching from Thunderbird to Kmail many years back. I am hoping that kmail gets fixed. It did pop-3 great under debian jessie.

 

John

 

 

 

 

On Wednesday, August 14, 2019 10:42:42 AM PDT René J.V. Bertin wrote:

> On Wednesday August 14 2019 17:30:38 Peter Humphrey wrote:

> > my mail on the local IMAP server instead of fetching it here. I've only

> > had

> > that in place since it became clear that KMail would never be fault free

> > with POP accounts.

>

> I'm always a bit surprised when I see people are actually still using that

> old, cumbersome protocol (POP3). I'd most likely configure the account(s)

> in question as external(s) in one of my gmail accounts, have gmail fetch

> the mail, filter it etc. and then read it over IMAP.

>

> If you already have a local IMAP server, why not use that for archiving

> (with the account configured to cache as little as possible in the akonadi

> db to prevent too much redundancy? The reason I use IMAP for archiving is

> that it's MUA-agnostic, and the actual folders are simple files I can hack

> into with vi if I have a reason to.

>

> R.

 

Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

René J.V. Bertin
On Wednesday August 14 2019 11:32:38 John White wrote:
>Consistency.  I am old and cumbered. I still use a foxpro varient of dbase   But I can
>search files with foxpro that are 30 years old.  Same with pop-3, though I made what

Not really a good comparison. The mail client hides most of the differences between POP3 and IMAP; I think that if you don't use folders on the server and only read you mail from a single machine you'll hardly notice a difference. IMAP should be a bit smoother, at least with a mail client that will only fetch the headers by default, unless you actually open a message for reading.

R.
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Randy Kramer
In reply to this post by René J.V. Bertin
On Wednesday, August 14, 2019 01:42:42 PM René J.V. Bertin wrote:
> If you already have a local IMAP server, why not use that for archiving
> (with the account configured to cache as little as possible in the akonadi
> db to prevent too much redundancy? The reason I use IMAP for archiving is
> that it's MUA-agnostic, and the actual folders are simple files I can hack
> into with vi if I have a reason to.

What is the format of those simple folders -- are they in mbox format by any
chance?

(I am a fervent user of both pop3 and mbox folders, and for lots of reasons
(including some software that I've developed and am continuing to develop
(using mbox files as the storage protocol) don't expect to ever willingly give
up either.)
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Randy Kramer
In reply to this post by René J.V. Bertin
On Wednesday, August 14, 2019 03:00:07 PM René J.V. Bertin wrote:
> Not really a good comparison. The mail client hides most of the differences
> between POP3 and IMAP; I think that if you don't use folders on the server
> and only read you mail from a single machine you'll hardly notice a
> difference. IMAP should be a bit smoother, at least with a mail client
> that will only fetch the headers by default, unless you actually open a
> message for reading.

Just to followup to my comment in a previous post (about pop3 and mbox file
format), I open almost all email messages for reading.  (The exceptions are
obvious spam, most of which is filtered out one way or another before I see it
(but I don't like the way gmail decides for me what is spam and what is not).)
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Jerome Yuzyk-2
In reply to this post by René J.V. Bertin
On Wednesday, August 14, 2019 11:42:42 AM MDT René J.V. Bertin wrote:

> On Wednesday August 14 2019 17:30:38 Peter Humphrey wrote:
> > my mail on the local IMAP server instead of fetching it here. I've only
> > had
> > that in place since it became clear that KMail would never be fault free
> > with POP accounts.
>
> I'm always a bit surprised when I see people are actually still using that
> old, cumbersome protocol (POP3). I'd most likely configure the account(s)
> in question as external(s) in one of my gmail accounts, have gmail fetch
> the mail, filter it etc. and then read it over IMAP.

I've been using KMail and several POP3 mailboxes (my ISP, and GMail and even
Hotmail) since KDE 2.something and I've never had the melt-downs I see from
IMAP users, especially after Akonadi got into the mix. I use maildir folders,
and that may help a lot. There are sometimes filtering problems, but that's
down to Akonadi's timing and filtering faults.




Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

René J.V. Bertin
In reply to this post by Randy Kramer
On Wednesday August 14 2019 17:40:49 [hidden email] wrote:

> What is the format of those simple folders -- are they in mbox format by any
> chance?

I think they are.

R.
Reply | Threaded
Open this post in threaded view
|

Re: All local folders lost in KMail

Randy Kramer
On Wednesday, August 14, 2019 08:08:50 PM René J.V. Bertin wrote:
> On Wednesday August 14 2019 17:40:49 [hidden email] wrote:
> > What is the format of those simple folders -- are they in mbox format by
> > any chance?
>
> I think they are.

Someday if it is convenient, maybe you could send one as an attachment so I
could look at it.  It could contain just a few (3 or 4) small, non-confidential
emails.

And no hurry, just something I'd like to know -- maybe at some point I'd
consider using an IMAP server for some part of my mail flow (assuming it uses
mbox files).

Thanks!
123