Discussion:
shotwell with 100k pics: somewhere between quite fast and too slow
oliver
2011-08-13 01:44:19 UTC
Permalink
Hello,

I had now used shotwell 0.10.1
and imported about 130.000 picture files
(jpeg, 720 x 576 pixel, about 10k per file).

I have not looked at the time in detail, but it needed
"the whole day", roughly estimated about 12 hours or so.

Thats much too slow.

I will test how long the creation of the thumbnails
with convert and shell will need. I would assume it's
much faster.


When scrolling in the overview of the files,
here shotwell performs very good.
Thats much better than I had in mind that f-spot behaves.
But maybe thats because I compared f-spot when it has not
already built the thumbnails.

But somehow shotwell seems to be much faster in that point.
(I could compare with f-spot maybe, when the thumbs are all created.
But f-spot is on a different machine, so the comparison can only
be roughly).


When I click on one of the thumbnails, it needs long to open
the picture (about 4 or 5 seconds).
Here f-spot seems to be quicker.

When I want to close that picture and go back to the overview
with thumbnails it needs even longer: about 38 seconds.

Nothing that really could be accepted.


In f-spot I have about 30,000 pictures, in the shotwell test I used
about 100,000.

But I would assume f-spot will perform better at 100,000 pictures.

Somehow it seems there must be done some work to optimize here.

Either the OO-way is too much performance hungry, or the algorithms
and/or datastructures in use could be optimized.
Maybe the handling of the database.

When I have opened one picture, then using the next/previous-arrows,
this performs quite well.

But switching between overview and single picture and back from single
picture to overview is unacceptable slow.


Maybe this can give the developers a hint on what might be necessary to look at.


Some issues are quite fast and others are very slow.
For a program with version number much below 1.0
this might be no surprise that not anything is perfect.

But I hope these issues will be addressed.


Ciao,
Oliver
oliver
2011-08-13 01:46:08 UTC
Permalink
Post by oliver
Hello,
I had now used shotwell 0.10.1
and imported about 130.000 picture files
(jpeg, 720 x 576 pixel, about 10k per file).
[...]

I did not do import with file-copy.
I only did import via links.

With copying I think import would would be even longer.
And some other issues might also behave differently.

Maybe I could test that at another time.


Ciao,
Oliver
oliver
2011-08-13 11:33:23 UTC
Permalink
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.

Is there an easy way for this?

How would I make a debugging version from shotwell?

(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)


Ciao,
Oliver
oliver
2011-08-13 11:47:44 UTC
Permalink
Is adding "-g" in the central makefile (VALA_CFLAGS) sufficient?
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
Andreas Brauchli
2011-08-13 11:57:35 UTC
Permalink
hi oliver

what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.

for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)

however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof

cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
oliver
2011-08-13 12:11:06 UTC
Permalink
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version.
Ah, yes.

Mixed it up.
Post by Andreas Brauchli
btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
OK, fine.
"-pg" ok, yes, now I remember.

So this will produce a profile-output that I can an analyse, as if I would
do it with C, I guess.
Post by Andreas Brauchli
however if you're not comfortable with c programming i would not advise
you to do so..
I'm comfortable with C-programming, but not often used the profiler
and so I forgot some details (and the switches), which I think will be possible to find
somewhere in the manpages. And nice, that you help here.

For me, profiling and debugging go hand-in-hand, hence I messed up both terms.
Post by Andreas Brauchli
not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
Yes. I used it - but not often, and it's many years ago.

So I forgot some switches-details.
Post by Andreas Brauchli
cheers and best of luck
Thanks.

Ciao,
Oliver
Adam Dingle
2011-08-14 07:45:39 UTC
Permalink
Oliver,

thanks for being brave enough to try Shotwell with 100K photos and for
reporting performance numbers in your previous email. As you've pointed
out, Shotwell doesn't yet scale nicely to libraries of this size. In
most of our testing at Yorba we haven't gone much beyond 10K photos,
though we've certainly had reports of individual users with 30K or
more. You're the first user I know of to try 100K. :)

And yes, we should make Shotwell more scalable to larger libraries.
I've created a ticket to track progress on this:

http://redmine.yorba.org/issues/3980

For profiling, I highly recomend sysprof:

http://sysprof.com/
http://live.gnome.org/Sysprof

The huge advantage of sysprof over other profilers (such as gprof) is
that you don't need a special profiling build of your program and that
your program runs at normal speed. You will need to build with debug
symbols, however, and you should also install debug symbols for glib,
libc and GTK. We've used sysprof for all our performance optimization
work in Shotwell so far.

To build Shotwell with debug symbols, run 'configure --debug' before you
run make.

adam
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
oliver
2011-08-14 16:54:50 UTC
Permalink
Hello Adam,

thanks for adding the 100k pics issue as ticket.

I may try also with sysprof later;
I did not used that tool so far.

So I will stay with gprof at least for a while,
which I already used, even it's a while ago.


Recompiling the libraries also for debugging/profiling
is the effort I try to avoid, as far as possible.


I already started debugging, but it seems that shotwell
is hanging, when compiled for profiling/debugging .

But no, it just finished while I write this mail.

So it just needs much time...
Just starting and closing took 14 minutes!


I will present my results in a seperate mail soon.


Ciao,
Oliver
Post by Adam Dingle
Oliver,
thanks for being brave enough to try Shotwell with 100K photos and
for reporting performance numbers in your previous email. As you've
pointed out, Shotwell doesn't yet scale nicely to libraries of this
size. In most of our testing at Yorba we haven't gone much beyond
10K photos, though we've certainly had reports of individual users
with 30K or more. You're the first user I know of to try 100K. :)
And yes, we should make Shotwell more scalable to larger libraries.
http://redmine.yorba.org/issues/3980
http://sysprof.com/
http://live.gnome.org/Sysprof
The huge advantage of sysprof over other profilers (such as gprof)
is that you don't need a special profiling build of your program and
that your program runs at normal speed. You will need to build with
debug symbols, however, and you should also install debug symbols
for glib, libc and GTK. We've used sysprof for all our performance
optimization work in Shotwell so far.
To build Shotwell with debug symbols, run 'configure --debug' before
you run make.
adam
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
Adam Dingle
2011-08-14 17:36:55 UTC
Permalink
That's why I recommend sysprof: there's no need to recompile libraries,
your program runs at normal speed and it's really easy to use as well.
I bet that if you install the sysprof package on your distribution, you
could have it working in less time that the 14 minutes it takes to start
Shotwell with gprof!

adam
Post by oliver
Hello Adam,
thanks for adding the 100k pics issue as ticket.
I may try also with sysprof later;
I did not used that tool so far.
So I will stay with gprof at least for a while,
which I already used, even it's a while ago.
Recompiling the libraries also for debugging/profiling
is the effort I try to avoid, as far as possible.
I already started debugging, but it seems that shotwell
is hanging, when compiled for profiling/debugging .
But no, it just finished while I write this mail.
So it just needs much time...
Just starting and closing took 14 minutes!
I will present my results in a seperate mail soon.
Ciao,
Oliver
Post by Adam Dingle
Oliver,
thanks for being brave enough to try Shotwell with 100K photos and
for reporting performance numbers in your previous email. As you've
pointed out, Shotwell doesn't yet scale nicely to libraries of this
size. In most of our testing at Yorba we haven't gone much beyond
10K photos, though we've certainly had reports of individual users
with 30K or more. You're the first user I know of to try 100K. :)
And yes, we should make Shotwell more scalable to larger libraries.
http://redmine.yorba.org/issues/3980
http://sysprof.com/
http://live.gnome.org/Sysprof
The huge advantage of sysprof over other profilers (such as gprof)
is that you don't need a special profiling build of your program and
that your program runs at normal speed. You will need to build with
debug symbols, however, and you should also install debug symbols
for glib, libc and GTK. We've used sysprof for all our performance
optimization work in Shotwell so far.
To build Shotwell with debug symbols, run 'configure --debug' before
you run make.
adam
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
oliver
2011-08-14 18:58:03 UTC
Permalink
Post by Adam Dingle
That's why I recommend sysprof: there's no need to recompile
libraries, your program runs at normal speed and it's really easy to
use as well.
Aha, ok. Sounds good. :-)

But it will run without configure's --debug?
(You mentioned that option...)
Post by Adam Dingle
I bet that if you install the sysprof package on your
distribution, you could have it working in less time that the 14
minutes it takes to start Shotwell with gprof!
[...]

hmhh.


Using -pg and/or -g option I found the problem, that
shotwell seem to hang.

Looked at it with strace, I found the following:

======================================================================
[...]
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
[...]
======================================================================

In one of the runs I had killed shotwell after about 40 minutes via kill(1).

Not sure if this also happens with sysprof.
I may try it, but the mentioned problem should not occure..


Ciao,
Oliver
Adam Dingle
2011-08-15 06:32:45 UTC
Permalink
Post by oliver
Post by Adam Dingle
That's why I recommend sysprof: there's no need to recompile
libraries, your program runs at normal speed and it's really easy to
use as well.
Aha, ok. Sounds good. :-)
But it will run without configure's --debug?
(You mentioned that option...)
Sysprof will run even if you haven't built your program with debug
symbols, but its results will be most informative if debug symbols are
present. Fortunately, it's easy to build Shotwell with debug symbols
(using --debug), and debug symbols for most common libraries (such as
glib and gtk) are easily available on major distributions.

adam
oliver
2011-08-15 09:23:39 UTC
Permalink
Post by Adam Dingle
Post by oliver
Post by Adam Dingle
That's why I recommend sysprof: there's no need to recompile
libraries, your program runs at normal speed and it's really easy to
use as well.
Aha, ok. Sounds good. :-)
But it will run without configure's --debug?
(You mentioned that option...)
Sysprof will run even if you haven't built your program with debug
symbols, but its results will be most informative if debug symbols
are present.
[...]

OK.

For gprof I read as a hint, it's better to avoid -pg and -g at the same time.

Ciao,
Oliver
oliver
2011-08-14 19:23:31 UTC
Permalink
nice tool,
but it only shows me percentage.

gprof shows me also number of calls.

Or maybe sysprof also can be customized?

BTW: I run it as root and it also shows kernel stuff.
Is it something like a dtrace-clone for Linux?


Ciao,
Oliver
Post by Adam Dingle
That's why I recommend sysprof: there's no need to recompile
libraries, your program runs at normal speed and it's really easy to
use as well. I bet that if you install the sysprof package on your
distribution, you could have it working in less time that the 14
minutes it takes to start Shotwell with gprof!
adam
Post by oliver
Hello Adam,
thanks for adding the 100k pics issue as ticket.
I may try also with sysprof later;
I did not used that tool so far.
So I will stay with gprof at least for a while,
which I already used, even it's a while ago.
Recompiling the libraries also for debugging/profiling
is the effort I try to avoid, as far as possible.
I already started debugging, but it seems that shotwell
is hanging, when compiled for profiling/debugging .
But no, it just finished while I write this mail.
So it just needs much time...
Just starting and closing took 14 minutes!
I will present my results in a seperate mail soon.
Ciao,
Oliver
Post by Adam Dingle
Oliver,
thanks for being brave enough to try Shotwell with 100K photos and
for reporting performance numbers in your previous email. As you've
pointed out, Shotwell doesn't yet scale nicely to libraries of this
size. In most of our testing at Yorba we haven't gone much beyond
10K photos, though we've certainly had reports of individual users
with 30K or more. You're the first user I know of to try 100K. :)
And yes, we should make Shotwell more scalable to larger libraries.
http://redmine.yorba.org/issues/3980
http://sysprof.com/
http://live.gnome.org/Sysprof
The huge advantage of sysprof over other profilers (such as gprof)
is that you don't need a special profiling build of your program and
that your program runs at normal speed. You will need to build with
debug symbols, however, and you should also install debug symbols
for glib, libc and GTK. We've used sysprof for all our performance
optimization work in Shotwell so far.
To build Shotwell with debug symbols, run 'configure --debug' before
you run make.
adam
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
Adam Dingle
2011-08-15 06:41:26 UTC
Permalink
Post by oliver
nice tool,
but it only shows me percentage.
gprof shows me also number of calls.
Right. sysprof reports only time, not a call count, because it's a
sampling profiler: it periodically stops the program and examines the
stack trace to see where the program is. It doesn't instrument the
code, so it has no idea how many times each function has been called.
This is why programs run virtually at normal speed under sysprof.

Of course, in optimizing code our most fundamental concern is reducing
execution time, and sysprof can reveal time bottlenecks even without
call counts.
Post by oliver
Or maybe sysprof also can be customized?
BTW: I run it as root and it also shows kernel stuff.
Yes: sysprof shows time used by all processes on the system. (This is
sometimes very useful, by the way.) When you use sysprof to profile
Shotwell, it's best to have no other CPU-intensive programs active at
the same time. Then in the sysprof output, most time will be spent
inside Shotwell; you can study that time and ignore the time spent in
other programs.
Post by oliver
Is it something like a dtrace-clone for Linux?
dtrace is more powerful in that it can insert probes (instrumentation
points) into the code. sysprof is really just a simple system-wide
sampling profiler. Fortunately we've found that for optimizing Shotwell
it works well enough.

adam
oliver
2011-08-14 14:28:38 UTC
Permalink
Hi Andreas,

I have not to add "-X -pg" in VALAFLAGS.
This does not really provide the "-pg" to the C-compiler.

Instead it must be used the following:
"-pg" added to VALA_CFLAGS


Ciao,
Oliver
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
oliver
2011-08-14 15:53:25 UTC
Permalink
"-pg" must also be added in VALA_LDFLAGS
Post by oliver
Hi Andreas,
I have not to add "-X -pg" in VALAFLAGS.
This does not really provide the "-pg" to the C-compiler.
"-pg" added to VALA_CFLAGS
Ciao,
Oliver
Post by Andreas Brauchli
hi oliver
what you want is probably not a debug version but a
profilable version. btw, it looks like the debug flag (-g) is passed by
default - at least if the packager didn't turn it off.
for profiling you can use gprof to do the job by passing -X -pg to valac
(VALAFLAGS in Makefile)
however if you're not comfortable with c programming i would not advise
you to do so.. not that you could break much but it could be
frustrating ;) at least you'd need to read up on how to use gprof
cheers and best of luck
andreas
Post by oliver
If it is possible to create a gdb-/debugging-version
of shotwell, and if this is easy by just adding
a switch to one makefile, I could try the same
procedure again, so that the bottleneck maybe
can be identified.
Is there an easy way for this?
How would I make a debugging version from shotwell?
(Or are those issues already addressed by the shotwell team?
Or is being able to handle about 100k pics not in the focus
of the shotwell team?)
Ciao,
Oliver
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
_______________________________________________
Shotwell mailing list
http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
oliver
2011-08-15 00:57:32 UTC
Permalink
Hello,

I did the proiling with 132324 imported pictures
(not copied to the shotwell "repository").

Used system: Arch-Linux on AMD64 (dual core) notebook/laptop.


General notes:
==============

1)
Sometimes shotwell seemed to hang, e.g.
when quitting it.

I found out that it not really hangs forever, but for a long time.
E.g. it ended after about 14 minutes, or in one test after 9 minutes...)

In case it "hangs" (temporarily, as mentioned above), strace gave me the
following output:


======================================================================
[...]
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
[...]
======================================================================


But shotwell sometimes finished immediately.

This seems only to occur, if profiling-switch is enabled.

I looked up the problem: it seems NOT to be shotwell-specific,
and so I already mentioned it on the linux kernel mailing list.
AFAIK, the ERESTARTNOINTR is not intended to pop up in user space.


2)
The slowness I talked about, also occured without these profiling-related "hanging".
I hope to have some time to lok at this issue again later.


3)
When starting shotwell, the progress bar goes up to 49% very fast.
After that, it stalls for a while, until the program opens.
When the stalling occurs, a lot of mprotect()/brk() is going on.


Results:
========

Because my last mail was too big for the mailinglist,
I put the results online; they are temporarily available here:

http://www.first.in-berlin.de/tmp/shotwell-gprof.tgz

Ciao,
Oliver

P.S.: The moderator of the list can cancel my last big mail,
because this one is the replacement for that big mail.

P.P.S.: The hanging-problem seems to be very unspecifc.
Sometimes it occurs, sometimes not, or at least it
seesm to hang not always the same amount of time.
I could not find out any regularity here.
oliver
2011-08-15 10:33:19 UTC
Permalink
On Mon, Aug 15, 2011 at 02:57:32AM +0200, oliver wrote:
[...]
Post by oliver
========
Because my last mail was too big for the mailinglist,
http://www.first.in-berlin.de/tmp/shotwell-gprof.tgz
[...]


Some sysprof-samples:
http://www.first.in-berlin.de/tmp/sysprof_shotwell_samples_1.tgz

This file contains:
sysprof_shotwell_only_open-close-of-picture_withoutstartquit.xml (sampled without program startup/quit)
sysprof_shotwell_open-andclose-pictures.xml (sampled just starting/quitting)
sysprof_shotwell_open-close_1.xml (sampled just starting/quitting)
sysprof_shotwell_openclose_2.xml (sampled just starting/quitting)
sysprof_shotwell_scrolling-overview.xml (sampled arbiritraily scrolling)

The scrolling issue is not critical, it is comparingly fast.
I just added it for "completeness".



Ciao,
Oliver

P.S.: When I first experimented with the sysprof,
I also had results with huge time spent in sqlite3.
But I could not reproduce that later.
(And I did not saved the results.)
oliver
2011-08-15 10:34:46 UTC
Permalink
Post by oliver
[...]
Post by oliver
========
Because my last mail was too big for the mailinglist,
http://www.first.in-berlin.de/tmp/shotwell-gprof.tgz
[...]
http://www.first.in-berlin.de/tmp/sysprof_shotwell_samples_1.tgz
sysprof_shotwell_only_open-close-of-picture_withoutstartquit.xml (sampled without program startup/quit)
sysprof_shotwell_open-andclose-pictures.xml (sampled just starting/quitting)
WRONG! it is: start, open-and-close-pic, quit.
[...]

Ciao,
Oliver
oliver
2011-08-15 10:08:26 UTC
Permalink
I reported about stalling many seconds, when
going back from detail view of a picture to
the picture-overview.

The following strace sequence is what I found:


=======================================================================================================
read(3, 0x25f1494, 4096) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=4, events=POLLIN}, {fd=3, events=POLLIN}, {fd=15, events=POLLIN|POLLPRI}, {fd=17, events=POLLIN|POLLPRI}, {fd=18, events=POLLIN|POLLPRI}, {fd=19, events=POLLIN|POLLPRI}, {fd=20, events=POLLIN}, {fd=21, events=POLLIN}], 8, 25) = 1 ([{fd=3, revents=POLLIN}])
read(3, "\2\tR5T5\16\0O\1\0\0\373\30\0\1\0\0\0\0G\1z\0F\1k\0\0\0\1\0", 4096) = 32
read(3, 0x25f1494, 4096) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\22\0\7\0\373\30\0\1\273\1\0\0\6\0\0\0 \4\4\0\1\0\0\0T5\16\0&\0\2\0"..., 36}, {NULL, 0}, {"", 0}], 3) = 36
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "\34\0S5\373\30\0\1\273\1\0\0V5\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 64
read(3, 0x25f1494, 4096) = -1 EAGAIN (Resource temporarily unavailable)

[
Here the shotwell stalls for a while.
When stalling is over, the following output comes...
]


brk(0x2329a000) = 0x2329a000
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLIN|POLLOUT}])
read(3, "\3\tT5\2275\16\0O\1\0\0\373\30\0\1\0\0\0\0G\1z\0F\1k\0\0\0\1\0"..., 4096) = 832
writev(3, [{"\2\0\4\0\373\30\0\1\0@\0\0(\32\0\1&\4\2\0\373\30\0\1", 24}, {NULL, 0}, {"", 0}], 3) = 24
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "\1\1V5\0\0\0\0O\1\0\0\0\0\0\0\0\1\2\2\377\0\363\1\0\0\0\0\0\0\0\0", 4096) = 32
read(3, 0x25f1494, 4096) = -1 EAGAIN (Resource temporarily unavailable)
read(3, 0x25f1494, 4096) = -1 EAGAIN (Resource temporarily unavailable)
writev(17, [{"GIOP\1\2\1\0\255\0\0\0", 12}, {"\250\203\376X\3\0\0\0\0\0j\2\34\0\0\0\0\0\0\0\335\300\370\210\232\265((\300+(("..., 173}], 2) = 185
poll([{fd=13, events=POLLIN}, {fd=17, events=POLLIN|POLLPRI}, {fd=18, events=POLLIN|POLLPRI}, {fd=19, events=POLLIN|POLLPRI}], 4, -1) = 1 ([{fd=17, revents=POLLIN}])
=======================================================================================================


Ciao,
Oliver
oliver
2011-08-14 19:51:55 UTC
Permalink
Hello,

I did the profiling with 132324 imported pictures
(not copied to the shotwell "repository").

All files are jpeg-files, 720 x 576 pixel, about 10k per file.


Used system: Arch-Linux on AMD64 (dual core) notebook/laptop.


General notes:
==============

1)
Sometimes shotwell seemed to hang, e.g.
when quitting it.

I found out that it not really hangs forever, but for a long time.
E.g. it ended after about 14 minutes, or in one test after 9 minutes...)

In case it "hangs" (temporarily, as mentioned above), strace gave me the
following output:


======================================================================
[...]
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fffdcc6ac8c) = ? ERESTARTNOINTR (To be restarted)
--- {si_signo=SIGPROF, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} (Profiling timer expired) ---
rt_sigreturn(0x1b) = 56
[...]
======================================================================


But shotwell sometimes finished immediately, without that temporarily hanging.


2)
When starting shotwell, the progress bar goes up to 49% very fast.
After that, it stalls for a while, until the program opens,
and shows the GUI.


Results:
========

See the attached files.


Trying around with sysprof at the moment.
Maybe more on that later.

Ciao,
Oliver

Loading...