Clicking on the trash can on the desktop brings up an instance of konqueror to view the trash:/ pseudo-directory. Unfortunately, it takes roughly twelve hours (!!) for the screen to be correctly populated and the progress bar in the bottom right hand corner to reach 100%. (It reaches 50% after about 20 minutes -- which I regard as still much too long simply to display the items in trash.)
For almost the entirety of the twelve hours, konqueror pins one CPU at 100%.
There are occasional bursts of LAN traffic as konqueror communicates with one of the two other machines that have mounted directories that contain some of the trash files; but most of the time there is no detectable LAN traffic and the CPU is simply pinned with no obvious work being done.
Attached is my ~/.trinity/share/config/trashrc file. I'm not sure how this file was populated, but it seems more or less consistent with the Properties | Trash Policy item associated with the Trash icon on the desktop. I note that I had to change the filename from "trashrc" to "trashrc.txt" (without changing the contents), as, bizarrely, gitea said that it could not accept files of "this type" when I tried to upload the original trashrc file.
I'm also attaching a screenshot of what konqueror looks like about 40 minutes after I clicked the desktop trash icon, just in case that offers you any hints.
Obviously, I'm happy to provide any other info that you might find useful, or try any (non-destructive) experiments that you might want me to carry out.
Steps to reproduce
Click the trash icon on the desktop.
Wait ~12 hours :-)
Screenshots
See the second of the attached files for an example.
<!--
This is a comment.
Please fill in the required fields below.
The comments provide instructions on how to do so.
Note: You do not need to remove comments.
-->
## Basic information
- TDE version: R14.1.0 DEVELOPMENT <!-- such as R14.0.12 - see tde-config -v -->
- Distribution: debian bullseye <!-- such as Debian Bullseye - see lsb_release -sd -->
- Hardware: amd64 <!-- amd64 / i386 / ppc64el / armhf / ... -->
<!--
Use SL/* labels to set the severity level.
Please do not set a milestone.
-->
## Description
Clicking on the trash can on the desktop brings up an instance of konqueror to view the trash:/ pseudo-directory. Unfortunately, it takes roughly twelve hours (!!) for the screen to be correctly populated and the progress bar in the bottom right hand corner to reach 100%. (It reaches 50% after about 20 minutes -- which I regard as still much too long simply to display the items in trash.)
For almost the entirety of the twelve hours, konqueror pins one CPU at 100%.
There are occasional bursts of LAN traffic as konqueror communicates with one of the two other machines that have mounted directories that contain some of the trash files; but most of the time there is no detectable LAN traffic and the CPU is simply pinned with no obvious work being done.
Attached is my ~/.trinity/share/config/trashrc file. I'm not sure how this file was populated, but it seems more or less consistent with the Properties | Trash Policy item associated with the Trash icon on the desktop. I note that I had to change the filename from "trashrc" to "trashrc.txt" (without changing the contents), as, bizarrely, gitea said that it could not accept files of "this type" when I tried to upload the original trashrc file.
I'm also attaching a screenshot of what konqueror looks like about 40 minutes after I clicked the desktop trash icon, just in case that offers you any hints.
Obviously, I'm happy to provide any other info that you might find useful, or try any (non-destructive) experiments that you might want me to carry out.
## Steps to reproduce
1. Click the trash icon on the desktop.
2. Wait ~12 hours :-)
## Screenshots
See the second of the attached files for an example.
Any chance you can try this in gdb and interrupt the waiting when there is no traffic and see where Konqueror is wasting the CPU time?
I can't reproduce the bug here, so will need some help if we want to troubleshoot this one.
Any chance you can try this in gdb and interrupt the waiting when there is no traffic and see where Konqueror is wasting the CPU time?
I can't reproduce the bug here, so will need some help if we want to troubleshoot this one.
Any chance you can try this in gdb and interrupt the waiting when there is no traffic and see where Konqueror is wasting the CPU time?
I can't reproduce the bug here, so will need some help if we want to troubleshoot this one.
I don't remember how it was possible to create .Trash bins on other directories, but I am sure that the application is memorizing these (./trinity/share/config/trashrc), so if they are on shared drived mounted locally and if you have too many of them, it can take some time, especially if network is not reliable.
I have two and lucily also can not reproduce. The first one is on NFS share and the second is the local Trash
/data-backup/DATA/.Trash-1000
/home/emanoil/.local/share/Trash
I am on TDE 14.1
> Any chance you can try this in gdb and interrupt the waiting when there is no traffic and see where Konqueror is wasting the CPU time?
> I can't reproduce the bug here, so will need some help if we want to troubleshoot this one.
I don't remember how it was possible to create .Trash bins on other directories, but I am sure that the application is memorizing these (./trinity/share/config/trashrc), so if they are on shared drived mounted locally and if you have too many of them, it can take some time, especially if network is not reliable.
I have two and lucily also can not reproduce. The first one is on NFS share and the second is the local Trash
/data-backup/DATA/.Trash-1000
/home/emanoil/.local/share/Trash
I am on TDE 14.1
Well, I'm glad that neither of you see the problem; I would hate to think that everyone basically can't look at the trash in konqueror.
The network is a gigabit home LAN, so it should be plenty fast enough for this task; there's rarely much other traffic on it. It is a bit of a puzzle why merely looking at the trash causes periods where a ton of traffic is being moved across the network for a few minutes at a time: surely it's just metadata that konqueror is grabbing, not tens of GB of actual trashed files. Anyway, most of the time while I'm waiting for konqueror to display trash:/ there is no obvious network traffic, so that's a minor question.
I'm not sure how to do what Michele wants. Tell me exactly what I need to do in order to produce something that would be of use, and I'll do it. I've used gdb, but only to look at core dumps from crashes or to step through my own programs -- neither of which is the case here, of course.
I just watched "top" off-and-on for about an hour after clicking on the desktop trash icon:
For the first few minutes, sshfs uses a lot of CPU cycles. That suggests that it's doing something with /home/n7dr/remote/shack20/.Trash-1000, which is the only directory mounted with sshfs. After a while, a lot of traffic flows across the LAN (although sshfs was heavily in use before any traffic flowed at all).
Then tdeio_trash takes over, using about 30% of a CPU. During this period, konqueror mostly uses almost no CPU, but it occasionally jumps to 100% use for a few seconds at a time.
Then, from about half an hour after I clicked the desktop icon, konqueror basically sits at 100%, and, as far as I can tell from the behaviour I've seen over the past few days, it will now sit at 100% for ~12 hours.
If I run "gdb attach and look at the stack trace when it's reached the stable state where konqueror is sitting more-or-less permanently at 100%, I get the attached file bt-konqueror.txt.
It doesn't look very helpful to me, but what do I know? :-)
If I attach to the tdeio_trash process at that time (which is running, but using almost no CPU), then I get the attached file bt-tdeio_trash.txt.
Suggestions as to how I can be more helpful would be gratefully received and acted on.
Well, I'm glad that neither of you see the problem; I would hate to think that everyone basically can't look at the trash in konqueror.
The network is a gigabit home LAN, so it should be plenty fast enough for this task; there's rarely much other traffic on it. It is a bit of a puzzle why merely looking at the trash causes periods where a ton of traffic is being moved across the network for a few minutes at a time: surely it's just metadata that konqueror is grabbing, not tens of GB of actual trashed files. Anyway, most of the time while I'm waiting for konqueror to display trash:/ there is no obvious network traffic, so that's a minor question.
I'm not sure how to do what Michele wants. Tell me exactly what I need to do in order to produce something that would be of use, and I'll do it. I've used gdb, but only to look at core dumps from crashes or to step through my own programs -- neither of which is the case here, of course.
I just watched "top" off-and-on for about an hour after clicking on the desktop trash icon:
For the first few minutes, sshfs uses a lot of CPU cycles. That suggests that it's doing something with /home/n7dr/remote/shack20/.Trash-1000, which is the only directory mounted with sshfs. After a while, a lot of traffic flows across the LAN (although sshfs was heavily in use before any traffic flowed at all).
Then tdeio_trash takes over, using about 30% of a CPU. During this period, konqueror mostly uses almost no CPU, but it occasionally jumps to 100% use for a few seconds at a time.
Then, from about half an hour after I clicked the desktop icon, konqueror basically sits at 100%, and, as far as I can tell from the behaviour I've seen over the past few days, it will now sit at 100% for ~12 hours.
If I run "gdb attach <PID of the konqueror process> and look at the stack trace when it's reached the stable state where konqueror is sitting more-or-less permanently at 100%, I get the attached file bt-konqueror.txt.
It doesn't look very helpful to me, but what do I know? :-)
If I attach to the tdeio_trash process at that time (which is running, but using almost no CPU), then I get the attached file bt-tdeio_trash.txt.
Suggestions as to how I can be more helpful would be gratefully received and acted on.
Hi Doc,
thanks for the Koqneuror bt, it is actually useful.
More questions:
are you using icons view or list view in Konqueror? I assume listview from the bt you attached.
how many items do you have in the Trash overall? That is after the 12 hours period, if you select all the stuff shown by Konqueror, how many entries are in the list?
do you have the konqueror filter plugin installed and do you have any pattern set before clicking on Trash?
if you make a copy of anything in the Trash into a local folder, then close Konqueror and reopen a new one and go to that folder, do you seem a similar behavior?
Hi Doc,
thanks for the Koqneuror bt, it is actually useful.
More questions:
1) are you using icons view or list view in Konqueror? I assume listview from the bt you attached.
2) how many items do you have in the Trash overall? That is after the 12 hours period, if you select all the stuff shown by Konqueror, how many entries are in the list?
3) do you have the konqueror filter plugin installed and do you have any pattern set before clicking on Trash?
4) if you make a copy of anything in the Trash into a local folder, then close Konqueror and reopen a new one and go to that folder, do you seem a similar behavior?
are you using icons view or list view in Konqueror? I assume listview from the bt you attached.
Yes, list view. A lot of the time the view is essentially blank, but it updates every now and then. Attached is a screenshot I just took of what it looks like at a random point while in the ~12-hour heavy CPU period. It's running now because of your next question:
how many items do you have in the Trash overall? That is after the 12
hours period, if you select all the stuff shown by Konqueror, how many
entries are in the list?
I will let you once it's finished.
do you have the konqueror filter plugin installed and do you have any pattern set before clicking on Trash?
I don't even know what a filter plugin is :-) So I imagine that the answer is "No".
Hmmm... I just took a screenshot and am attaching it as home-konqueror2.png. Maybe that visible menu means that I do have the filter plugin installed?
I'm pretty sure that I didn't install it manually, but perhaps it's automatically installed when running TDE on debian.
if you make a copy of anything in the Trash into a local folder, then
close Konqueror and reopen a new one and go to that folder, do you seem a
similar behavior?
I don't want to do anything that might mess with the 12-hour job that's currently running. But once that's completed and I have the answer to your second question, I'll do as you suggest and let you know the result.
Response to questions:
> are you using icons view or list view in Konqueror? I assume listview from the bt you attached.
Yes, list view. A lot of the time the view is essentially blank, but it updates every now and then. Attached is a screenshot I just took of what it looks like at a random point while in the ~12-hour heavy CPU period. It's running now because of your next question:
> how many items do you have in the Trash overall? That is after the 12
hours period, if you select all the stuff shown by Konqueror, how many
entries are in the list?
I will let you once it's finished.
> do you have the konqueror filter plugin installed and do you have any pattern set before clicking on Trash?
I don't even know what a filter plugin is :-) So I imagine that the answer is "No".
Hmmm... I just took a screenshot and am attaching it as home-konqueror2.png. Maybe that visible menu means that I do have the filter plugin installed?
I'm pretty sure that I didn't install it manually, but perhaps it's automatically installed when running TDE on debian.
> if you make a copy of anything in the Trash into a local folder, then
close Konqueror and reopen a new one and go to that folder, do you seem a
similar behavior?
I don't want to do anything that might mess with the 12-hour job that's currently running. But once that's completed and I have the answer to your second question, I'll do as you suggest and let you know the result.
I'm becoming increasingly suspicious of the sshfs mount somehow being at the root of all this.
The 12-hour job is still running, so I can't do as much investigating as I would like (by, for example, unmounting that filesystem). But it seems that if I trash a file on any other filesystem (by using the "Move to Trash" menu item in konqueror), it is correctly trashed immediately -- at least, it disappears from the ordinary directory in which it was located.
But when I tried to trash a file on the /home/n7dr/remote/shack20 filesystem, what happened is that a "progress dialog" popped up, sitting at 0%. And now, a couple of hours later, the progress dialog is still visible, and still on 0%, and the file I tried to trash, which was in no way special, is still sitting in the original directory. So that suggests that, for that filesystem, something is blocking whatever process it is that actually performs the action of moving a file to trash.
So my theory, pending further investigation, is that this is all something to do with the calls that konqueror is using via sshfs to access the sshfs-mounted filesystem.
When it is safe to do so (which might not be until tomorrow, now) I will temporarily unmount /home/n7dr/remote/shack20 and see what happens when I click on the "Trash" desktop icon. I reckon that there's a good chance that konqueror will display the trash for all the remaining mounted filesystems promptly and correctly.
Stay tuned :-)
I don't know what the stack looks like, but I imagine that it goes something like: konqueror <-> tdeio_trash <-> probably something else here related to the OS and filesystems <-> sshfs <-> ssh. Although the symptom of the problem is at the konqueror level, I'm beginning to think that the actual problem lies at, perhaps, the sshfs layer. It certainly seems to me that it's increasingly unlikely that it's a konqueror issue, and konqueror is using so much CPU only because it's continually trying to do something with which one of the lower layers is, for some reason, having difficulty. But of course, maybe I'm completely wrong :-) I often am.
Hmmm...
I'm becoming increasingly suspicious of the sshfs mount somehow being at the root of all this.
The 12-hour job is still running, so I can't do as much investigating as I would like (by, for example, unmounting that filesystem). But it seems that if I trash a file on any other filesystem (by using the "Move to Trash" menu item in konqueror), it is correctly trashed immediately -- at least, it disappears from the ordinary directory in which it was located.
But when I tried to trash a file on the /home/n7dr/remote/shack20 filesystem, what happened is that a "progress dialog" popped up, sitting at 0%. And now, a couple of hours later, the progress dialog is still visible, and still on 0%, and the file I tried to trash, which was in no way special, is still sitting in the original directory. So that suggests that, for that filesystem, something is blocking whatever process it is that actually performs the action of moving a file to trash.
So my theory, pending further investigation, is that this is all something to do with the calls that konqueror is using via sshfs to access the sshfs-mounted filesystem.
When it is safe to do so (which might not be until tomorrow, now) I will temporarily unmount /home/n7dr/remote/shack20 and see what happens when I click on the "Trash" desktop icon. I reckon that there's a good chance that konqueror will display the trash for all the remaining mounted filesystems promptly and correctly.
Stay tuned :-)
I don't know what the stack looks like, but I imagine that it goes something like: konqueror <-> tdeio_trash <-> probably something else here related to the OS and filesystems <-> sshfs <-> ssh. Although the symptom of the problem is at the konqueror level, I'm beginning to think that the actual problem lies at, perhaps, the sshfs layer. It certainly seems to me that it's increasingly unlikely that it's a konqueror issue, and konqueror is using so much CPU only because it's continually trying to do something with which one of the lower layers is, for some reason, having difficulty. But of course, maybe I'm completely wrong :-) I often am.
I don't even know what a filter plugin is :-) So I imagine that the answer is "No".
You have the plugin installed, it is visible in the screenshot (top right corner of Konqueror. It is part of tdeaddons, so that's probably how you installed it. You don't have any filter string there, so I think we can safely exclude this from the picture.
I'm becoming increasingly suspicious of the sshfs mount somehow being at the root of all this.
You have lot of stuff in Trash, at least 58K files based on your screenshot, probably more. So there are two possible reasons for the problem.
Konqueror gets incredibly slow when handling so many files in a view
ssh (or something else) is causing the delay.
Once you are done with the current operation, we can do some tests to further narrow the problem.
a. to test 1., copy all the Trash entries into a local disk. Then close Konqueror, open it again and go to that folder. If it still takes 12h or so, then the problem is most likely Konqueror
b. try to access the ssh Trash folder using mc from command line: if it still takes 12h, then the problem is most likely ssh related (or anything on the way)
> I don't even know what a filter plugin is :-) So I imagine that the answer is "No".
You have the plugin installed, it is visible in the screenshot (top right corner of Konqueror. It is part of tdeaddons, so that's probably how you installed it. You don't have any filter string there, so I think we can safely exclude this from the picture.
> I'm becoming increasingly suspicious of the sshfs mount somehow being at the root of all this.
You have lot of stuff in Trash, at least 58K files based on your screenshot, probably more. So there are two possible reasons for the problem.
1. Konqueror gets incredibly slow when handling so many files in a view
2. ssh (or something else) is causing the delay.
Once you are done with the current operation, we can do some tests to further narrow the problem.
a. to test 1., copy all the Trash entries into a local disk. Then close Konqueror, open it again and go to that folder. If it still takes 12h or so, then the problem is most likely Konqueror
b. try to access the ssh Trash folder using `mc` from command line: if it still takes 12h, then the problem is most likely ssh related (or anything on the way)
how many items do you have in the Trash overall? That is after the 12
hours period, if you select all the stuff shown by Konqueror, how many
entries are in the list?
konqueror says: 76647 items, 85.3 GB total.
That looks to be awfully big, so I investigated a bit.
Max size of trash on filesystems listed by "Properties for Trash" tab on the desktop icon:
So, obviously, there is indeed a problem with the sshfs-mounted filesystem: the maximum size is supposed to be 18.18 GB, which sounds about right, as it's a 2TB mdraid system; but /zd1/usgs/.Trash-1000 is occupying 84.3 GB of disk space.
Looking a bit more closely:
/home/n7dr/remote/shack20/.Trash-1000/files : 84.1 GB, 77900 files, 734 subdirectories
So that greatly exceeds the 1% number in the trashrc file.
That is, even though trashrc says that the filesystem trash should be controlled by:
it seems that whatever process is supposed to cleanup the trash when it gets too large is failing to do so, at least for that filesystem. (I don't know what that process is, or how often it runs. I guess that it's supposed to run once per day, but I don't know how to check that it actually does so, nor at what time it runs.)
Presumably there's a way to manually force the trash cleanup process to run, which might be an interesting thing to do, especially if it can be done in some kind of debug or verbose mode, but I don't know how to do that either. (I don't know much really, do I?)
I note that it take essentially no time at all to open a new konqueror session and navigate manually to /home/n7dr/remote/shack20/.Trash-1000, and then look at the contents. The same is true if I use mc. And also true if I simply go to that directory and look at it using the bash command line.
So it looks to me (but what do I know?) that tdeio_trash is the most likely culprit, or, failing that, something else to do with konqueror; but it doesn't look likely at this point that it's anything to do with sshfs or ssh itself.
Responses, now that konqueror has completed:
> how many items do you have in the Trash overall? That is after the 12
hours period, if you select all the stuff shown by Konqueror, how many
entries are in the list?
konqueror says: 76647 items, 85.3 GB total.
That looks to be awfully big, so I investigated a bit.
Max size of trash on filesystems listed by "Properties for Trash" tab on the desktop icon:
/home/n7dr : 16.99 GB
/tmp : 28.90 GB
/tmp/bu : 28.80 GB
/home/n7dr : 35.44 GB
/home/n7dr/remote/zserver : 17.87 GB
/zd1 : 16.88 GB
/zd1/rbn : 17.44 GB
/zd1/usgs : 16.99 GB
/zd1/public-logs : 17.30 GB
/zd1/backups : 16.88 GB
/home/n7dr/backups/backup : 35.44 GB
/bu.enc/n7dr : 35.44 GB
/home/n7dr/remote/shack20 : 18.18 GB
So those should be the maximum sizes on each filesystem (don't ask me why /home/n7dr appears twice, and with different numbers; I have no idea).
Actual sizes of the trash directories listed in trashrc:
[/home/n7dr/.Trash-1000] : 0
[/home/n7dr/.local/share/Trash] : 2.6 GB
[/home/n7dr/backups/backup/.Trash-1000] : 0
[/home/n7dr/remote/shack20/.Trash-1000] : 84.3 GB
[/home/n7dr/remote/zserver/.Trash-1000] : 315.9 kB
[/tmp/.Trash-1000] : 9.6 kB
[/tmp/bu/.Trash-1000] : 218 MB
[/zd1/.Trash-1000] : 0
[/zd1/backups/.Trash-1000] : 11.2 kB
[/zd1/public-logs/.Trash-1000] : 582.7 kB
[/zd1/rbn/.Trash-1000] : 0
[/zd1/usgs/.Trash-1000] : 0
So, obviously, there is indeed a problem with the sshfs-mounted filesystem: the maximum size is supposed to be 18.18 GB, which sounds about right, as it's a 2TB mdraid system; but /zd1/usgs/.Trash-1000 is occupying 84.3 GB of disk space.
Looking a bit more closely:
/home/n7dr/remote/shack20/.Trash-1000/files : 84.1 GB, 77900 files, 734 subdirectories
So that greatly exceeds the 1% number in the trashrc file.
That is, even though trashrc says that the filesystem trash should be controlled by:
[/home/n7dr/remote/shack20/.Trash-1000]
Days=7
FixedSize=500
FixedSizeUnit=2
LimitReachedAction=1
Percent=1
SizeLimitType=0
UseSizeLimit=true
UseTimeLimit=false
it seems that whatever process is supposed to cleanup the trash when it gets too large is failing to do so, at least for that filesystem. (I don't know what that process is, or how often it runs. I guess that it's supposed to run once per day, but I don't know how to check that it actually does so, nor at what time it runs.)
Presumably there's a way to manually force the trash cleanup process to run, which might be an interesting thing to do, especially if it can be done in some kind of debug or verbose mode, but I don't know how to do that either. (I don't know much really, do I?)
I note that it take essentially no time at all to open a new konqueror session and navigate manually to /home/n7dr/remote/shack20/.Trash-1000, and then look at the contents. The same is true if I use mc. And also true if I simply go to that directory and look at it using the bash command line.
So it looks to me (but what do I know?) that tdeio_trash is the most likely culprit, or, failing that, something else to do with konqueror; but it doesn't look likely at this point that it's anything to do with sshfs or ssh itself.
This will provide some info on where tdeio_trash is spending its time.
If the ~/tdeio_trash_profiling.data grows too big, stops the test earlier. If the file is not that big, you can run the test for longer time and collect more info
Thanks for the detailed report. So this raises a few points:
1. why the trash settings are not applied to a remote ssh disk
2. why there are repeated entries in trashrc for the same folder and with different sizes (I suspect the first may be an older settings perhaps)
3. why it takes ages to see the Trash from Konqueror, when going to the same remote folder is pretty fast instead.
4. option improvement: trash GUI should support different settings for different disks
Can you try this additional test?
1. start Konqueror and go to Trash
2. get the pid of tdeio_trash
3. from CLI run: `perf record -g -p `pidof tdeio_trasht` -o ~/tdeio_trash_profiling.data`
4. let it record for some time (half an hour maybe?)
5. stop `perf` with ctrl-C
6. from CLI run the following commands and paste here the initial part of their output
```
perf report --stdio -i ~/tdeio_trash_profiling.data
perf report --stdio --sort=dso -g none -i ~/tdeio_trash_profiling.data
perf report --stdio -g none -i ~/tdeio_trash_profiling.data
perf report --stdio -g -i ~/tdeio_trash_profiling.data
```
This will provide some info on where tdeio_trash is spending its time.
If the ~/tdeio_trash_profiling.data grows too big, stops the test earlier. If the file is not that big, you can run the test for longer time and collect more info
You're getting more than you asked for :-) I hope that that's OK.
I gathered the performance data twice: once for half an hour or so immediately after starting konqueror, and then again, separately, for half an hour or so when everything reached the steady state where konqueror is consuming 100% of a CPU, but nothing else much seems to be happening.
I coded the file names so that information from the first of those runs contains the word "start", and information from the second of the runs contains the word "steady" (because it's more or less a steady state).
These are the two files: naked_start and naked_steady.
You're getting more than you asked for :-) I hope that that's OK.
I gathered the performance data twice: once for half an hour or so immediately after starting konqueror, and then again, separately, for half an hour or so when everything reached the steady state where konqueror is consuming 100% of a CPU, but nothing else much seems to be happening.
I coded the file names so that information from the first of those runs contains the word "start", and information from the second of the runs contains the word "steady" (because it's more or less a steady state).
> perf report --stdio -i ~/tdeio_trash_profiling.data
These are the two files: prof_start and prof_steady.
> perf report --stdio --sort=dso -g none -i ~/tdeio_trash_profiling.data
These are the two files: sorted_start and sorted_steady.
> perf report --stdio -g none -i ~/tdeio_trash_profiling.data
These are the two files: none_start and none_steady.
> perf report --stdio -g -i ~/tdeio_trash_profiling.data
These are the two files: naked_start and naked_steady.
Hmmm, it isn't clear that the files were actually included in the post. Can you see them? If not, maybe I need to rename them by adding ".txt" or something similar to their names.
Hmmm, it isn't clear that the files were actually included in the post. Can you see them? If not, maybe I need to rename them by adding ".txt" or something similar to their names.
Or maybe there's a size limit and they exceed it. Let me put them somewhere for you.
OK, you should now be able to download the directory:
https://www.adrive.com/public/y2Xeur/tde-trash
It contains the eight perf output files.
Thanks for the files, I have downloaded them and saved for later analysis.
Another question, because I realized I may have misunderstood some of the previous comment and made the wrong deduction/assumption and you kast comment raised a doubt.
Which one is the process using 100% CPU time? tdeio_trash or Konqueror?
If it is Konqueror, could you repeating the profiling using Konqueror pid instead of the tdeio_trash pid? Doing at the steady state would suffice.
Also, regardless of the previous point, could you also do a system-wide profiling when in steady (100% CPU) state? It's teh same procedure but without the -p pidof of process thing when recording. This will show the relative load of Konqueror vs. tdeio-trash vs. ssh. If possible, try to do as little as you can during the recording period, to minimize other unnecessary stuff.
Thanks for the files, I have downloaded them and saved for later analysis.
Another question, because I realized I may have misunderstood some of the previous comment and made the wrong deduction/assumption and you kast comment raised a doubt.
Which one is the process using 100% CPU time? tdeio_trash or Konqueror?
If it is Konqueror, could you repeating the profiling using Konqueror pid instead of the tdeio_trash pid? Doing at the steady state would suffice.
Also, regardless of the previous point, could you also do a system-wide profiling when in steady (100% CPU) state? It's teh same procedure but without the `-p pidof of process` thing when recording. This will show the relative load of Konqueror vs. tdeio-trash vs. ssh. If possible, try to do as little as you can during the recording period, to minimize other unnecessary stuff.
Which one is the process using 100% CPU time? tdeio_trash or Konqueror?
konqueror
If it is Konqueror, could you repeating the profiling using Konqueror
pid instead of the tdeio_trash pid? Doing at the steady state would
suffice.
OK; I will do that, probably today; if not today, then tomorrow.
system-wide profiling
OK; it might be tricky to find a time when nothing much else is happening on the machine, but I'll try to do the best I can
> Which one is the process using 100% CPU time? tdeio_trash or Konqueror?
konqueror
> If it is Konqueror, could you repeating the profiling using Konqueror
pid instead of the tdeio_trash pid? Doing at the steady state would
suffice.
OK; I will do that, probably today; if not today, then tomorrow.
> system-wide profiling
OK; it might be tricky to find a time when nothing much else is happening on the machine, but I'll try to do the best I can
OK; I will do that, probably today; if not today, then tomorrow.
Thanks and no rush. In any case investigation will be done after R14.1.0 is out, so take your time.
konqueror
Ah! I had misunderstood indeed.
> OK; I will do that, probably today; if not today, then tomorrow.
Thanks and no rush. In any case investigation will be done after R14.1.0 is out, so take your time.
> konqueror
Ah! I had misunderstood indeed.
MicheleC
added this to the R14.1.x milestone 1 year ago
I have added the output files from these commands:
[ZB:perf] perf report --stdio -i konqueror.data > prof_konq
[ZB:perf] perf report --stdio --sort=dso -g none -i konqueror.data > sorted_konq
[ZB:perf] perf report --stdio -g none -i konqueror.data > none_konq
[ZB:perf] perf report --stdio -g -i konqueror.data > naked_konq
to the directory: https://www.adrive.com/public/y2Xeur/tde-trash
They are the perf data from konqueror when it is in the steady state.
As you probably expected, the konq perf data are considerably larger than the tdeio_trash data.
I will run the system-wide profiling when I get the chance (I know you said that there was no hurry, but I find that it's best to do these things while my mind is on them :-) )
I have added the output files from these commands:
[ZB:perf] perf report --stdio -i konqueror.data > prof_konq
[ZB:perf] perf report --stdio --sort=dso -g none -i konqueror.data > sorted_konq
[ZB:perf] perf report --stdio -g none -i konqueror.data > none_konq
[ZB:perf] perf report --stdio -g -i konqueror.data > naked_konq
to the directory:
https://www.adrive.com/public/y2Xeur/tde-trash
They are the perf data from konqueror when it is in the steady state.
As you probably expected, the konq perf data are considerably larger than the tdeio_trash data.
I will run the system-wide profiling when I get the chance (I know you said that there was no hurry, but I find that it's best to do these things while my mind is on them :-) )
I ran the system profile for ten minutes instead of the thirty minutes (more or less) that I used for the other profiles. I figured that ten minutes would probably be enough. If you discover that ten minutes isn't long enough (when you eventually have a chance to look at the files), let me know and I'll run it for longer.
I have added the system profiling files as well to the directory:
https://www.adrive.com/public/y2Xeur/tde-trash
I ran the system profile for ten minutes instead of the thirty minutes (more or less) that I used for the other profiles. I figured that ten minutes would probably be enough. If you discover that ten minutes isn't long enough (when you eventually have a chance to look at the files), let me know and I'll run it for longer.
The new files are the output from:
[ZB:perf] perf report --stdio -i system.data > prof_system
[ZB:perf] perf report --stdio --sort=dso -g none -i system.data > sorted_system
[ZB:perf] perf report --stdio -g none -i system.data > none_system
[ZB:perf] perf report --stdio -g -i system.data > naked_system
Thanks for the files. At first sight, konqueror listview and tqt3 seems to be the ones using lot of CPU time. We will need to find out why the same does not happen if you navigate to the big folder directly, but I suspect that is where the problem is, because navigating the Trash will use the trdeio-trash slave while navigating a folder won't.
Once I come back to this bug for investigation, I may have more questions :-)
Thanks for the files. At first sight, konqueror listview and tqt3 seems to be the ones using lot of CPU time. We will need to find out why the same does not happen if you navigate to the big folder directly, but I suspect that is where the problem is, because navigating the Trash will use the trdeio-trash slave while navigating a folder won't.
Once I come back to this bug for investigation, I may have more questions :-)
I just tried to send a couple of small files in the directory "/home/n7dr/remote/shack20/drlog/2022/AA CW" to trash (using "Move to Trash" in konqueror). Basically, it just hangs.
Since I need to be able to send stuff to trash from that filesystem, I'm going to try running:
/usr/local/bin/autotrash -d 35 -T "/home/n7dr/remote/shack20/.Trash-1000"
to try to at least clean things up in that Trash directory.
It may make all the problems go away, which would be a bit irritating, as it might mean that we never understand what was happening, but I think I have to run that risk, as I really need to be able to trash things on that filesystem. I do have a backup of the files and info files, so, if necessary, I could try restoring those directories if we need to try to make it all fail again.
Anyway, I'll run that autotrash command and see what happens.
I just tried to send a couple of small files in the directory "/home/n7dr/remote/shack20/drlog/2022/AA CW" to trash (using "Move to Trash" in konqueror). Basically, it just hangs.
Since I need to be able to send stuff to trash from that filesystem, I'm going to try running:
/usr/local/bin/autotrash -d 35 -T "/home/n7dr/remote/shack20/.Trash-1000"
to try to at least clean things up in that Trash directory.
It may make all the problems go away, which would be a bit irritating, as it might mean that we never understand what was happening, but I think I have to run that risk, as I really need to be able to trash things on that filesystem. I do have a backup of the files and info files, so, if necessary, I could try restoring those directories if we need to try to make it all fail again.
Anyway, I'll run that autotrash command and see what happens.
And, as I feared, that fixes the reported problem. trash:/ now displays in about 30 seconds or so, without any obvious issues.
If you want me to try restoring the trash from backup at some point, I can do that, and see if that makes it fail again.
And, as I feared, that fixes the reported problem. trash:/ now displays in about 30 seconds or so, without any obvious issues.
If you want me to try restoring the trash from backup at some point, I can do that, and see if that makes it fail again.
If you want me to try restoring the trash from backup at some point, I can do that, and see if that makes it fail again.
I may ask for this in future, if we need to do a new test.
Thanks for the update so far. This bug is in my TODO list, so I will come back to it at some point.
> If you want me to try restoring the trash from backup at some point, I can do that, and see if that makes it fail again.
I may ask for this in future, if we need to do a new test.
Thanks for the update so far. This bug is in my TODO list, so I will come back to it at some point.
A couple of months of ordinary use after "fixing" the problem on this machine as described above, the problem is back. From this I conclude that my original problem was not just a rare, random event, but points to a real issue.
FYI:
A couple of months of ordinary use after "fixing" the problem on this machine as described above, the problem is back. From this I conclude that my original problem was not just a rare, random event, but points to a real issue.
Basic information
Description
Clicking on the trash can on the desktop brings up an instance of konqueror to view the trash:/ pseudo-directory. Unfortunately, it takes roughly twelve hours (!!) for the screen to be correctly populated and the progress bar in the bottom right hand corner to reach 100%. (It reaches 50% after about 20 minutes -- which I regard as still much too long simply to display the items in trash.)
For almost the entirety of the twelve hours, konqueror pins one CPU at 100%.
There are occasional bursts of LAN traffic as konqueror communicates with one of the two other machines that have mounted directories that contain some of the trash files; but most of the time there is no detectable LAN traffic and the CPU is simply pinned with no obvious work being done.
Attached is my ~/.trinity/share/config/trashrc file. I'm not sure how this file was populated, but it seems more or less consistent with the Properties | Trash Policy item associated with the Trash icon on the desktop. I note that I had to change the filename from "trashrc" to "trashrc.txt" (without changing the contents), as, bizarrely, gitea said that it could not accept files of "this type" when I tried to upload the original trashrc file.
I'm also attaching a screenshot of what konqueror looks like about 40 minutes after I clicked the desktop trash icon, just in case that offers you any hints.
Obviously, I'm happy to provide any other info that you might find useful, or try any (non-destructive) experiments that you might want me to carry out.
Steps to reproduce
Screenshots
See the second of the attached files for an example.
Any chance you can try this in gdb and interrupt the waiting when there is no traffic and see where Konqueror is wasting the CPU time?
I can't reproduce the bug here, so will need some help if we want to troubleshoot this one.
I don't remember how it was possible to create .Trash bins on other directories, but I am sure that the application is memorizing these (./trinity/share/config/trashrc), so if they are on shared drived mounted locally and if you have too many of them, it can take some time, especially if network is not reliable.
I have two and lucily also can not reproduce. The first one is on NFS share and the second is the local Trash
/data-backup/DATA/.Trash-1000
/home/emanoil/.local/share/Trash
I am on TDE 14.1
Well, I'm glad that neither of you see the problem; I would hate to think that everyone basically can't look at the trash in konqueror.
The network is a gigabit home LAN, so it should be plenty fast enough for this task; there's rarely much other traffic on it. It is a bit of a puzzle why merely looking at the trash causes periods where a ton of traffic is being moved across the network for a few minutes at a time: surely it's just metadata that konqueror is grabbing, not tens of GB of actual trashed files. Anyway, most of the time while I'm waiting for konqueror to display trash:/ there is no obvious network traffic, so that's a minor question.
I'm not sure how to do what Michele wants. Tell me exactly what I need to do in order to produce something that would be of use, and I'll do it. I've used gdb, but only to look at core dumps from crashes or to step through my own programs -- neither of which is the case here, of course.
I just watched "top" off-and-on for about an hour after clicking on the desktop trash icon:
For the first few minutes, sshfs uses a lot of CPU cycles. That suggests that it's doing something with /home/n7dr/remote/shack20/.Trash-1000, which is the only directory mounted with sshfs. After a while, a lot of traffic flows across the LAN (although sshfs was heavily in use before any traffic flowed at all).
Then tdeio_trash takes over, using about 30% of a CPU. During this period, konqueror mostly uses almost no CPU, but it occasionally jumps to 100% use for a few seconds at a time.
Then, from about half an hour after I clicked the desktop icon, konqueror basically sits at 100%, and, as far as I can tell from the behaviour I've seen over the past few days, it will now sit at 100% for ~12 hours.
If I run "gdb attach and look at the stack trace when it's reached the stable state where konqueror is sitting more-or-less permanently at 100%, I get the attached file bt-konqueror.txt.
It doesn't look very helpful to me, but what do I know? :-)
If I attach to the tdeio_trash process at that time (which is running, but using almost no CPU), then I get the attached file bt-tdeio_trash.txt.
Suggestions as to how I can be more helpful would be gratefully received and acted on.
Hi Doc,
thanks for the Koqneuror bt, it is actually useful.
More questions:
Response to questions:
Yes, list view. A lot of the time the view is essentially blank, but it updates every now and then. Attached is a screenshot I just took of what it looks like at a random point while in the ~12-hour heavy CPU period. It's running now because of your next question:
I will let you once it's finished.
I don't even know what a filter plugin is :-) So I imagine that the answer is "No".
Hmmm... I just took a screenshot and am attaching it as home-konqueror2.png. Maybe that visible menu means that I do have the filter plugin installed?
I'm pretty sure that I didn't install it manually, but perhaps it's automatically installed when running TDE on debian.
I don't want to do anything that might mess with the 12-hour job that's currently running. But once that's completed and I have the answer to your second question, I'll do as you suggest and let you know the result.
Hmmm...
I'm becoming increasingly suspicious of the sshfs mount somehow being at the root of all this.
The 12-hour job is still running, so I can't do as much investigating as I would like (by, for example, unmounting that filesystem). But it seems that if I trash a file on any other filesystem (by using the "Move to Trash" menu item in konqueror), it is correctly trashed immediately -- at least, it disappears from the ordinary directory in which it was located.
But when I tried to trash a file on the /home/n7dr/remote/shack20 filesystem, what happened is that a "progress dialog" popped up, sitting at 0%. And now, a couple of hours later, the progress dialog is still visible, and still on 0%, and the file I tried to trash, which was in no way special, is still sitting in the original directory. So that suggests that, for that filesystem, something is blocking whatever process it is that actually performs the action of moving a file to trash.
So my theory, pending further investigation, is that this is all something to do with the calls that konqueror is using via sshfs to access the sshfs-mounted filesystem.
When it is safe to do so (which might not be until tomorrow, now) I will temporarily unmount /home/n7dr/remote/shack20 and see what happens when I click on the "Trash" desktop icon. I reckon that there's a good chance that konqueror will display the trash for all the remaining mounted filesystems promptly and correctly.
Stay tuned :-)
I don't know what the stack looks like, but I imagine that it goes something like: konqueror <-> tdeio_trash <-> probably something else here related to the OS and filesystems <-> sshfs <-> ssh. Although the symptom of the problem is at the konqueror level, I'm beginning to think that the actual problem lies at, perhaps, the sshfs layer. It certainly seems to me that it's increasingly unlikely that it's a konqueror issue, and konqueror is using so much CPU only because it's continually trying to do something with which one of the lower layers is, for some reason, having difficulty. But of course, maybe I'm completely wrong :-) I often am.
You have the plugin installed, it is visible in the screenshot (top right corner of Konqueror. It is part of tdeaddons, so that's probably how you installed it. You don't have any filter string there, so I think we can safely exclude this from the picture.
You have lot of stuff in Trash, at least 58K files based on your screenshot, probably more. So there are two possible reasons for the problem.
Once you are done with the current operation, we can do some tests to further narrow the problem.
a. to test 1., copy all the Trash entries into a local disk. Then close Konqueror, open it again and go to that folder. If it still takes 12h or so, then the problem is most likely Konqueror
b. try to access the ssh Trash folder using
mcfrom command line: if it still takes 12h, then the problem is most likely ssh related (or anything on the way)Responses, now that konqueror has completed:
konqueror says: 76647 items, 85.3 GB total.
That looks to be awfully big, so I investigated a bit.
Max size of trash on filesystems listed by "Properties for Trash" tab on the desktop icon:
/home/n7dr : 16.99 GB
/tmp : 28.90 GB
/tmp/bu : 28.80 GB
/home/n7dr : 35.44 GB
/home/n7dr/remote/zserver : 17.87 GB
/zd1 : 16.88 GB
/zd1/rbn : 17.44 GB
/zd1/usgs : 16.99 GB
/zd1/public-logs : 17.30 GB
/zd1/backups : 16.88 GB
/home/n7dr/backups/backup : 35.44 GB
/bu.enc/n7dr : 35.44 GB
/home/n7dr/remote/shack20 : 18.18 GB
So those should be the maximum sizes on each filesystem (don't ask me why /home/n7dr appears twice, and with different numbers; I have no idea).
Actual sizes of the trash directories listed in trashrc:
[/home/n7dr/.Trash-1000] : 0
[/home/n7dr/.local/share/Trash] : 2.6 GB
[/home/n7dr/backups/backup/.Trash-1000] : 0
[/home/n7dr/remote/shack20/.Trash-1000] : 84.3 GB
[/home/n7dr/remote/zserver/.Trash-1000] : 315.9 kB
[/tmp/.Trash-1000] : 9.6 kB
[/tmp/bu/.Trash-1000] : 218 MB
[/zd1/.Trash-1000] : 0
[/zd1/backups/.Trash-1000] : 11.2 kB
[/zd1/public-logs/.Trash-1000] : 582.7 kB
[/zd1/rbn/.Trash-1000] : 0
[/zd1/usgs/.Trash-1000] : 0
So, obviously, there is indeed a problem with the sshfs-mounted filesystem: the maximum size is supposed to be 18.18 GB, which sounds about right, as it's a 2TB mdraid system; but /zd1/usgs/.Trash-1000 is occupying 84.3 GB of disk space.
Looking a bit more closely:
/home/n7dr/remote/shack20/.Trash-1000/files : 84.1 GB, 77900 files, 734 subdirectories
So that greatly exceeds the 1% number in the trashrc file.
That is, even though trashrc says that the filesystem trash should be controlled by:
[/home/n7dr/remote/shack20/.Trash-1000]
Days=7
FixedSize=500
FixedSizeUnit=2
LimitReachedAction=1
Percent=1
SizeLimitType=0
UseSizeLimit=true
UseTimeLimit=false
it seems that whatever process is supposed to cleanup the trash when it gets too large is failing to do so, at least for that filesystem. (I don't know what that process is, or how often it runs. I guess that it's supposed to run once per day, but I don't know how to check that it actually does so, nor at what time it runs.)
Presumably there's a way to manually force the trash cleanup process to run, which might be an interesting thing to do, especially if it can be done in some kind of debug or verbose mode, but I don't know how to do that either. (I don't know much really, do I?)
I note that it take essentially no time at all to open a new konqueror session and navigate manually to /home/n7dr/remote/shack20/.Trash-1000, and then look at the contents. The same is true if I use mc. And also true if I simply go to that directory and look at it using the bash command line.
So it looks to me (but what do I know?) that tdeio_trash is the most likely culprit, or, failing that, something else to do with konqueror; but it doesn't look likely at this point that it's anything to do with sshfs or ssh itself.
Thanks for the detailed report. So this raises a few points:
Can you try this additional test?
perf record -g -ppidof tdeio_trasht-o ~/tdeio_trash_profiling.dataperfwith ctrl-CThis will provide some info on where tdeio_trash is spending its time.
If the ~/tdeio_trash_profiling.data grows too big, stops the test earlier. If the file is not that big, you can run the test for longer time and collect more info
You're getting more than you asked for :-) I hope that that's OK.
I gathered the performance data twice: once for half an hour or so immediately after starting konqueror, and then again, separately, for half an hour or so when everything reached the steady state where konqueror is consuming 100% of a CPU, but nothing else much seems to be happening.
I coded the file names so that information from the first of those runs contains the word "start", and information from the second of the runs contains the word "steady" (because it's more or less a steady state).
These are the two files: prof_start and prof_steady.
These are the two files: sorted_start and sorted_steady.
These are the two files: none_start and none_steady.
These are the two files: naked_start and naked_steady.
Hmmm, it isn't clear that the files were actually included in the post. Can you see them? If not, maybe I need to rename them by adding ".txt" or something similar to their names.
Or maybe there's a size limit and they exceed it. Let me put them somewhere for you.
OK, you should now be able to download the directory:
https://www.adrive.com/public/y2Xeur/tde-trash
It contains the eight perf output files.
Thanks for the files, I have downloaded them and saved for later analysis.
Another question, because I realized I may have misunderstood some of the previous comment and made the wrong deduction/assumption and you kast comment raised a doubt.
Which one is the process using 100% CPU time? tdeio_trash or Konqueror?
If it is Konqueror, could you repeating the profiling using Konqueror pid instead of the tdeio_trash pid? Doing at the steady state would suffice.
Also, regardless of the previous point, could you also do a system-wide profiling when in steady (100% CPU) state? It's teh same procedure but without the
-p pidof of processthing when recording. This will show the relative load of Konqueror vs. tdeio-trash vs. ssh. If possible, try to do as little as you can during the recording period, to minimize other unnecessary stuff.konqueror
OK; I will do that, probably today; if not today, then tomorrow.
OK; it might be tricky to find a time when nothing much else is happening on the machine, but I'll try to do the best I can
Thanks and no rush. In any case investigation will be done after R14.1.0 is out, so take your time.
Ah! I had misunderstood indeed.
I have added the output files from these commands:
[ZB:perf] perf report --stdio -i konqueror.data > prof_konq
[ZB:perf] perf report --stdio --sort=dso -g none -i konqueror.data > sorted_konq
[ZB:perf] perf report --stdio -g none -i konqueror.data > none_konq
[ZB:perf] perf report --stdio -g -i konqueror.data > naked_konq
to the directory:
https://www.adrive.com/public/y2Xeur/tde-trash
They are the perf data from konqueror when it is in the steady state.
As you probably expected, the konq perf data are considerably larger than the tdeio_trash data.
I will run the system-wide profiling when I get the chance (I know you said that there was no hurry, but I find that it's best to do these things while my mind is on them :-) )
I have added the system profiling files as well to the directory:
https://www.adrive.com/public/y2Xeur/tde-trash
I ran the system profile for ten minutes instead of the thirty minutes (more or less) that I used for the other profiles. I figured that ten minutes would probably be enough. If you discover that ten minutes isn't long enough (when you eventually have a chance to look at the files), let me know and I'll run it for longer.
The new files are the output from:
[ZB:perf] perf report --stdio -i system.data > prof_system
[ZB:perf] perf report --stdio --sort=dso -g none -i system.data > sorted_system
[ZB:perf] perf report --stdio -g none -i system.data > none_system
[ZB:perf] perf report --stdio -g -i system.data > naked_system
Thanks for the files. At first sight, konqueror listview and tqt3 seems to be the ones using lot of CPU time. We will need to find out why the same does not happen if you navigate to the big folder directly, but I suspect that is where the problem is, because navigating the Trash will use the trdeio-trash slave while navigating a folder won't.
Once I come back to this bug for investigation, I may have more questions :-)
I just tried to send a couple of small files in the directory "/home/n7dr/remote/shack20/drlog/2022/AA CW" to trash (using "Move to Trash" in konqueror). Basically, it just hangs.
Since I need to be able to send stuff to trash from that filesystem, I'm going to try running:
/usr/local/bin/autotrash -d 35 -T "/home/n7dr/remote/shack20/.Trash-1000"
to try to at least clean things up in that Trash directory.
It may make all the problems go away, which would be a bit irritating, as it might mean that we never understand what was happening, but I think I have to run that risk, as I really need to be able to trash things on that filesystem. I do have a backup of the files and info files, so, if necessary, I could try restoring those directories if we need to try to make it all fail again.
Anyway, I'll run that autotrash command and see what happens.
And, as I feared, that fixes the reported problem. trash:/ now displays in about 30 seconds or so, without any obvious issues.
If you want me to try restoring the trash from backup at some point, I can do that, and see if that makes it fail again.
Incidentally, the autotrash command ran to completion quite quickly, perhaps a minute or so.
I may ask for this in future, if we need to do a new test.
Thanks for the update so far. This bug is in my TODO list, so I will come back to it at some point.
FYI:
A couple of months of ordinary use after "fixing" the problem on this machine as described above, the problem is back. From this I conclude that my original problem was not just a rare, random event, but points to a real issue.