Bug #889084 “Java memory usage” : Bugs : Syncany

Revision history for this message

Philipp C. Heckel (binwiederhier) wrote on 2011-11-20:

#1

Very true unfortunately! Will be on the radar for potential releases...

Changed in syncany:
status:	New → Confirmed

Revision history for this message

Adam Porter (alphapapa) wrote on 2012-02-17:

#2

Not that it matters, but the use of Java ensures that I (and many others, I'm sure) will never use this software. Java is basically unsuitable for desktop software. It's incredibly slow and bloated. I can't afford to give hundreds of megs of RAM to an app for syncing files on all of my systems. And the startup time for the VM is crazy. Dropbox is not Free Software but it's mostly efficient and lightweight.

Everyone wants a Free Software Dropbox clone, but it seems like the most promising projects are all using things like Mono and Java. I don't get it. I wish someone talented and practical like Linus would just sit down and write one in C (or even Python) and be done with it.

Revision history for this message

João Pedro Afonso Oliveira de Almeida (jpoa) wrote on 2012-02-17: Re: [Syncany-team] [Bug 889084] Re: Java memory usage

#3

I can assure you well coded Java can beat C++. I emphasize the well coded.

The days of Java being slower every time are gone. You can't just expect to
put some code together and get high performance, if you want that stick to
VB.

And yes, this includes backend servers at big corporations, handling quite
a large amount of connections simultaneously (with db integration, disk
access and so on).

Now Mono is another story :)

On Fri, Feb 17, 2012 at 5:40 AM, Adam Porter <email address hidden>wrote:

> Not that it matters, but the use of Java ensures that I (and many
> others, I'm sure) will never use this software. Java is basically
> unsuitable for desktop software. It's incredibly slow and bloated. I
> can't afford to give hundreds of megs of RAM to an app for syncing files
> on all of my systems. And the startup time for the VM is crazy.
> Dropbox is not Free Software but it's mostly efficient and lightweight.
>
> Everyone wants a Free Software Dropbox clone, but it seems like the most
> promising projects are all using things like Mono and Java. I don't get
> it. I wish someone talented and practical like Linus would just sit
> down and write one in C (or even Python) and be done with it.
>
> --
> You received this bug notification because you are a member of Syncany
> Team, which is subscribed to Syncany.
> https://bugs.launchpad.net/bugs/889084
>
> Title:
> Java memory usage
>
> Status in Syncany:
> Confirmed
>
> Bug description:
> Just running deamon, java process use lot of memory (near to 1Go!)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/syncany/+bug/889084/+subscriptions
>
> --
> Mailing list: https://launchpad.net/~syncany-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~syncany-team
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message

Adam Porter (alphapapa) wrote on 2012-02-20:

#4

For enterprise use, that's different. My point is that, for desktop/laptop/netbook/whatever client use, the VM overhead makes it totally unsuitable. Just the startup time is horrendous, but the extra memory usage is unbearable.

Revision history for this message

João Pedro Afonso Oliveira de Almeida (jpoa) wrote on 2012-02-20:

#5

Well, ok, the startup time is slower, thats true, as well as an increase on
memory usage (objects use implicitly more than in other languages), but I
don't think its that critical...

Skype is worse for me actually...

On Mon, Feb 20, 2012 at 8:16 AM, Adam Porter <email address hidden>wrote:

> For enterprise use, that's different. My point is that, for
> desktop/laptop/netbook/whatever client use, the VM overhead makes it
> totally unsuitable. Just the startup time is horrendous, but the extra
> memory usage is unbearable.
>
> --
> You received this bug notification because you are a member of Syncany
> Team, which is subscribed to Syncany.
> https://bugs.launchpad.net/bugs/889084
>
> Title:
> Java memory usage
>
> Status in Syncany:
> Confirmed
>
> Bug description:
> Just running deamon, java process use lot of memory (near to 1Go!)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/syncany/+bug/889084/+subscriptions
>
> --
> Mailing list: https://launchpad.net/~syncany-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~syncany-team
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message

ClashTheBunny (spam-mason) wrote on 2012-02-20:

#6

Syncany is a beast. Syncany is not just Dropbox or SparkleShare.
Syncany actually does encryption of the data, they built the
infrastructure for block chunking with deduplication, they have
multiple backends. This is a feature full program; that isn't a
simple thing. If you use ZFS with deduplication you need many
gigabytes of memory and it's not doing encryption. Add the
compression and you get even worse memory usage. People who have
storage arrays of a few terabytes have systems with 16GB of RAM and
the ARC Cache alone takes up 6 GB or so during normal usage. This is
a fully featured program. ZFS in software almost. It's not something
simple like Dropbox.

If you can come up with a closer analog than ZFS that uses much less
RAM and has better startup times, I would love to see it. I've been
looking for a while and still haven't found an optimal storage
platform. If my Android phone, a memory limited device, can handle
many complicated tasks and run fairly quickly, I don't think it's Java
that's the problem. It's a combination of memory hungry, processor
intensive features and bugs that need fixing.

This is opensource. If you have such a strong opinion on what
language should be used for this project, you are free to take it and
port it. You have that freedom. With opensource, you are never in a
position where you can demand or complain. Come with data, like a
profile that shows where the code spends a lot of time or wastes a lot
of memory. Come with a patch. We all want this to be better. One of
the points of opensource is acknowledging that we aren't smart enough
ourselves and we need help with this.

Finally, when the new code is pushed, there will be better
opportunities to target specific parts. If you want a daemon written
in C or Python, it will be easier. This architecture issue is on the
TODO and will make it much easier to tackle this type of problem.
This way it will be easier to think of Syncany less as a specific
program, like AIM, but more like a protocol, like XMPP. When a
Syncany client is written for iOS, it will be written in Objective-C,
when it's written for Android, it will be in Java, when it's the
backend, there may be a couple of different backend implementations
with different advantages. We will, hopefully soon, have that
freedom.

R. Mason
<email address hidden>

Syncany is a beast.  Syncany is not just Dropbox or SparkleShare.
Syncany actually does encryption of the data, they built the
infrastructure for block chunking with deduplication, they have
multiple backends.  This is a feature full program; that isn't a
simple thing.  If you use ZFS with deduplication you need many
gigabytes of memory and it's not doing encryption.  Add the
compression and you get even worse memory usage.  People who have
storage arrays of a few terabytes have systems with 16GB of RAM and
the ARC Cache alone takes up 6 GB or so during normal usage.  This is
a fully featured program.  ZFS in software almost.  It's not something
simple like Dropbox.

If you can come up with a closer analog than ZFS that uses much less
RAM and has better startup times, I would love to see it.  I've been
looking for a while and still haven't found an optimal storage
platform.  If my Android phone, a memory limited device, can handle
many complicated tasks and run fairly quickly, I don't think it's Java
that's the problem.  It's a combination of memory hungry, processor
intensive features and bugs that need fixing.

This is opensource.  If you have such a strong opinion on what
language should be used for this project, you are free to take it and
port it.  You have that freedom.  With opensource, you are never in a
position where you can demand or complain.  Come with data, like a
profile that shows where the code spends a lot of time or wastes a lot
of memory.  Come with a patch.  We all want this to be better.  One of
the points of opensource is acknowledging that we aren't smart enough
ourselves and we need help with this.

Finally, when the new code is pushed, there will be better
opportunities to target specific parts.  If you want a daemon written
in C or Python, it will be easier.  This architecture issue is on the
TODO and will make it much easier to tackle this type of problem.
This way it will be easier to think of Syncany less as a specific
program, like AIM, but more like a protocol, like XMPP.  When a
Syncany client is written for iOS, it will be written in Objective-C,
when it's written for Android, it will be in Java, when it's the
backend, there may be a couple of different backend implementations
with different advantages.  We will, hopefully soon, have that
freedom.

R. Mason
spam@mason.ch

Revision history for this message

João Pedro Afonso Oliveira de Almeida (jpoa) wrote on 2012-02-20:

#7

Now now there, ZFS is on its own league :D And setting up a nice array is
an art ;)

Plus, I don't think Syncany can ensure data integrity all the way like ZFS
does.

On Mon, Feb 20, 2012 at 10:03 AM, Spam <email address hidden> wrote:

> Syncany is a beast. Syncany is not just Dropbox or SparkleShare.
> Syncany actually does encryption of the data, they built the
> infrastructure for block chunking with deduplication, they have
> multiple backends. This is a feature full program; that isn't a
> simple thing. If you use ZFS with deduplication you need many
> gigabytes of memory and it's not doing encryption. Add the
> compression and you get even worse memory usage. People who have
> storage arrays of a few terabytes have systems with 16GB of RAM and
> the ARC Cache alone takes up 6 GB or so during normal usage. This is
> a fully featured program. ZFS in software almost. It's not something
> simple like Dropbox.
>
> If you can come up with a closer analog than ZFS that uses much less
> RAM and has better startup times, I would love to see it. I've been
> looking for a while and still haven't found an optimal storage
> platform. If my Android phone, a memory limited device, can handle
> many complicated tasks and run fairly quickly, I don't think it's Java
> that's the problem. It's a combination of memory hungry, processor
> intensive features and bugs that need fixing.
>
> This is opensource. If you have such a strong opinion on what
> language should be used for this project, you are free to take it and
> port it. You have that freedom. With opensource, you are never in a
> position where you can demand or complain. Come with data, like a
> profile that shows where the code spends a lot of time or wastes a lot
> of memory. Come with a patch. We all want this to be better. One of
> the points of opensource is acknowledging that we aren't smart enough
> ourselves and we need help with this.
>
> Finally, when the new code is pushed, there will be better
> opportunities to target specific parts. If you want a daemon written
> in C or Python, it will be easier. This architecture issue is on the
> TODO and will make it much easier to tackle this type of problem.
> This way it will be easier to think of Syncany less as a specific
> program, like AIM, but more like a protocol, like XMPP. When a
> Syncany client is written for iOS, it will be written in Objective-C,
> when it's written for Android, it will be in Java, when it's the
> backend, there may be a couple of different backend implementations
> with different advantages. We will, hopefully soon, have that
> freedom.
>
> R. Mason
> <email address hidden>
>
> --
> Mailing list: https://launchpad.net/~syncany-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~syncany-team
> More help : https://help.launchpad.net/ListHelp
>

Now now there, ZFS is on its own league :D And setting up a nice array is
an art ;)

Plus, I don't think Syncany can ensure data integrity all the way like ZFS
does.

On Mon, Feb 20, 2012 at 10:03 AM, Spam <spam@mason.ch> wrote:

> Syncany is a beast.  Syncany is not just Dropbox or SparkleShare.
> Syncany actually does encryption of the data, they built the
> infrastructure for block chunking with deduplication, they have
> multiple backends.  This is a feature full program; that isn't a
> simple thing.  If you use ZFS with deduplication you need many
> gigabytes of memory and it's not doing encryption.  Add the
> compression and you get even worse memory usage.  People who have
> storage arrays of a few terabytes have systems with 16GB of RAM and
> the ARC Cache alone takes up 6 GB or so during normal usage.  This is
> a fully featured program.  ZFS in software almost.  It's not something
> simple like Dropbox.
>
> If you can come up with a closer analog than ZFS that uses much less
> RAM and has better startup times, I would love to see it.  I've been
> looking for a while and still haven't found an optimal storage
> platform.  If my Android phone, a memory limited device, can handle
> many complicated tasks and run fairly quickly, I don't think it's Java
> that's the problem.  It's a combination of memory hungry, processor
> intensive features and bugs that need fixing.
>
> This is opensource.  If you have such a strong opinion on what
> language should be used for this project, you are free to take it and
> port it.  You have that freedom.  With opensource, you are never in a
> position where you can demand or complain.  Come with data, like a
> profile that shows where the code spends a lot of time or wastes a lot
> of memory.  Come with a patch.  We all want this to be better.  One of
> the points of opensource is acknowledging that we aren't smart enough
> ourselves and we need help with this.
>
> Finally, when the new code is pushed, there will be better
> opportunities to target specific parts.  If you want a daemon written
> in C or Python, it will be easier.  This architecture issue is on the
> TODO and will make it much easier to tackle this type of problem.
> This way it will be easier to think of Syncany less as a specific
> program, like AIM, but more like a protocol, like XMPP.  When a
> Syncany client is written for iOS, it will be written in Objective-C,
> when it's written for Android, it will be in Java, when it's the
> backend, there may be a couple of different backend implementations
> with different advantages.  We will, hopefully soon, have that
> freedom.
>
> R. Mason
> spam@mason.ch
>
> --
> Mailing list: https://launchpad.net/~syncany-team
> Post to     : syncany-team@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~syncany-team
> More help   : https://help.launchpad.net/ListHelp
>

Revision history for this message

Adam Porter (alphapapa) wrote on 2012-02-20:

#8

Hm, I don't think ZFS is a relevant comparison. It's a filesystem, and one basically intended for server use, anyway.

That Java is a pig for desktop apps is well known and self-evident. Among the ones I've used over the years are Azureus, NeverNote, CrashPlan, and others that I can't think of at the moment. And they are all slow to load and use much more memory than equivalent programs written in C (or even Python). They also hold on to memory that they no longer need, whereas otherwise it could release it (if properly written).

It might be ok for 8-core, 8 GB machines that are rarely rebooted--but there are a wide range of devices that such software needs to run on. Dropbox uses about 70 MB on my 2 GB netbook--and even that's pushing it. It's not practical for services that run all the time in the background to use hundreds of megs of memory. Pretty soon you don't have any left to run actual apps (including ever-growing hogs like Firefox).

Yes, yes: open-source, Free Software, freedom-to-fork, spare-time, show-us-the-code, patches-welcome, blah, blah, blah. I'm no newbie to FOSS--I've heard all this before. I'm just making a few points about why Java is not a good choice for this kind of project, and why it will hinder the project in the long run. Time will tell. Feel free to ignore everything I say. No offense intended.

Syncany

Java memory usage

Bug Description

Other bug subscribers

Remote bug watches