High data transfer and cpu activity while idling

The repeating output is quite a normal behavior:
- It is polling for changes at the storage.
- And sending NOPs to the python client so that it doesnt die (this is ugly, but necessary!)

It periodically retrieves a list of files from the storage (S3) - every 15 seconds - to check if there is something new. The high bandwidth is caused by the amount of files in the bucket... But 200kb/s is a lot.

I just took a quick look at the implementation of the S3StorageManager and it seems that there's a way to fix this. I'm on it :-D

Thanks for reporting.

Another problemthat I just realized is, that it used all the 2000 free PUT, COPY, POST, or LIST requests that are included in the AWS free monthly tier in less than a day.

Thanks for looking into this.

I dont think that this is an issue. 400 of them were for the PUT
requests for the chunks (200MB/0.5MB = 400 chunks); the rest was the
polling. I suppose if we'd make the polling interval configurable,
that'd be a nice thing... So that it would make 1 request per minute,
or even 1 per 5 minutes...

On Wed, Jun 15, 2011 at 9:21 PM, zweiblum <email address hidden> wrote:
> Another problemthat I just realized is, that it used all the 2000 free
> PUT, COPY, POST, or LIST requests that are included in the AWS free
> monthly tier in less than a day.
>
> Thanks for looking into this.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/797823
>
> Title:
>  High data transfer and cpu activity while idling
>
> Status in Syncany:
>  Confirmed
>
> Bug description:
>  After having uploaded about 200mb from my syncany folder to a remote
>  (amazon s3, encrypted) location and restarting syncany, it produces a
>  lot of download traffic (about 250kb/s, nearly all the bandwidth of my
>  DSL connection) and some cpu activity.
>
>  $ uname -a
>  Linux jo-notebook 2.6.38-ARCH #1 SMP PREEMPT Mon Jun 6 22:49:29 CEST 2011 x86_64 Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz GenuineIntel GNU/Linux
>
>  $ java -version
>  java version "1.6.0_22"
>  OpenJDK Runtime Environment (IcedTea6 1.10.2) (ArchLinux-6.b22_1.10.2-1-x86_64)
>  OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>
>  $ python --version
>  Python 3.2
>
>  Syncany is repeating the following terminal output:
>
>  FO     : 4. No local changes. Skipping step upload.
>  11-06-15 12:49:03 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : DONE WITH PERIODIC UPDATE CHECK ...
>  11-06-15 12:49:06 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : STARTING PERIODIC UPDATE CHECK ...
>  11-06-15 12:49:07 | jo-notebook | PythonScript         | TrayServRead       | INFO     : time 1308156547.44 - last req 1308156543.01 > timeout 6
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Connected to native server on port 50,023
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Sent request {"request":"NopRequest"}. Waiting for response ...
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Client connected.
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Received request: {"request":"NopRequest"}
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Received response: null
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Doing nothing. That's what I do best :-)
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Sending response: OK
>  11-06-15 12:49:12 | jo-notebook | PythonScript         | TrayServRead       | INFO     : time 1308156552.45 - last req 1308156548.01 > timeout 6
>  11-06-15 12:49:12 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : repository has not changed locally. No need to upload.
>  11-06-15 12:...

I think polling is not good, maybe a future plan could be to write the "server" component, so each can "push" each other the changes? If a change on client happens, it notifies the server, if a change on server happens (from another client), server pushes the notification to the client and client re-syncs.

Polling is the only way to make it work with arbitrary storage types;
you cannot run any software on S3, FTP, ....

On Thu, Jun 16, 2011 at 9:37 AM, Fastguy <email address hidden> wrote:
> I think polling is not good, maybe a future plan could be to write the
> "server" component, so each can "push" each other the changes? If a
> change on client happens, it notifies the server, if a change on server
> happens (from another client), server pushes the notification to the
> client and client re-syncs.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/797823
>
> Title:
>  High data transfer and cpu activity while idling
>
> Status in Syncany:
>  Confirmed
>
> Bug description:
>  After having uploaded about 200mb from my syncany folder to a remote
>  (amazon s3, encrypted) location and restarting syncany, it produces a
>  lot of download traffic (about 250kb/s, nearly all the bandwidth of my
>  DSL connection) and some cpu activity.
>
>  $ uname -a
>  Linux jo-notebook 2.6.38-ARCH #1 SMP PREEMPT Mon Jun 6 22:49:29 CEST 2011 x86_64 Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz GenuineIntel GNU/Linux
>
>  $ java -version
>  java version "1.6.0_22"
>  OpenJDK Runtime Environment (IcedTea6 1.10.2) (ArchLinux-6.b22_1.10.2-1-x86_64)
>  OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>
>  $ python --version
>  Python 3.2
>
>  Syncany is repeating the following terminal output:
>
>  FO     : 4. No local changes. Skipping step upload.
>  11-06-15 12:49:03 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : DONE WITH PERIODIC UPDATE CHECK ...
>  11-06-15 12:49:06 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : STARTING PERIODIC UPDATE CHECK ...
>  11-06-15 12:49:07 | jo-notebook | PythonScript         | TrayServRead       | INFO     : time 1308156547.44 - last req 1308156543.01 > timeout 6
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Connected to native server on port 50,023
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Sent request {"request":"NopRequest"}. Waiting for response ...
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Client connected.
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Received request: {"request":"NopRequest"}
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Received response: null
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Doing nothing. That's what I do best :-)
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Sending response: OK
>  11-06-15 12:49:12 | jo-notebook | PythonScript         | TrayServRead       | INFO     : time 1308156552.45 - last req 1308156548.01 > timeout 6
>  11-06-15 12:49:12 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : repository has not changed locally. No need to upload.
>  11-06-15 12:49:12 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : 2. Dow...

I've implemented this (the arbitary checking inverval) in lp:~isak-karlsson/syncany/syncany-remote-watch-interval

By the way. Could not the bandwith usage be reduced by simply having a 0 byte file that indicates wheter or not the remote location is changed?

What the remotewatcher is doing is:
- retrieve a file list of all files (--> later it will be just
"list(prefix=update-...")
- check the file names of the update-files
- if there are unknown update files, download the,

so all we do is listing the files; that bandwidth that uses strongly depends on:
1. whether the storage type supports listing files with a prefix; e.g.
list all files with "update-...." prefix
2. if it does not support the prefix-stuff, it depends on the number of chunks.

as I said, right now it asks for a complete list once in the
beginning. but i could imagine something like that:

updates = getList("update-...")

if (new updates available) {
   list = getList() // complete list
else
   list = cachedList

So in fact the 0-byte file idea is exactly what I'm doing; except that
the files have content :-D
Do you have other ideas?

On Sat, Jun 18, 2011 at 1:49 PM, isakkarlsson <email address hidden> wrote:
> By the way. Could not the bandwith usage be reduced by simply having a 0
> byte file that indicates wheter or not the remote location is changed?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/797823
>
> Title:
>  High data transfer and cpu activity while idling
>
> Status in Syncany:
>  Confirmed
>
> Bug description:
>  After having uploaded about 200mb from my syncany folder to a remote
>  (amazon s3, encrypted) location and restarting syncany, it produces a
>  lot of download traffic (about 250kb/s, nearly all the bandwidth of my
>  DSL connection) and some cpu activity.
>
>  $ uname -a
>  Linux jo-notebook 2.6.38-ARCH #1 SMP PREEMPT Mon Jun 6 22:49:29 CEST 2011 x86_64 Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz GenuineIntel GNU/Linux
>
>  $ java -version
>  java version "1.6.0_22"
>  OpenJDK Runtime Environment (IcedTea6 1.10.2) (ArchLinux-6.b22_1.10.2-1-x86_64)
>  OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>
>  $ python --version
>  Python 3.2
>
>  Syncany is repeating the following terminal output:
>
>  FO     : 4. No local changes. Skipping step upload.
>  11-06-15 12:49:03 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : DONE WITH PERIODIC UPDATE CHECK ...
>  11-06-15 12:49:06 | jo-notebook | RemoteWatcher        | RemoteWatcher      | INFO     : STARTING PERIODIC UPDATE CHECK ...
>  11-06-15 12:49:07 | jo-notebook | PythonScript         | TrayServRead       | INFO     : time 1308156547.44 - last req 1308156543.01 > timeout 6
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Connected to native server on port 50,023
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Sent request {"request":"NopRequest"}. Waiting for response ...
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Client connected.
>  11-06-15 12:49:08 | jo-notebook | PythonScript         | TrayServRead       | INFO     : Received request: {"request":"NopRequest"}
>  11-06-15 12:49:08 | jo-notebook | LinuxNativeClient    | NativeNOPThread    | INFO     : Received response: null
>  11-06-15 12:49:08 | jo-notebook | PythonScript        ...