Modernize txaws.client to use twisted.web.client.Agent

As part of this, I think the receiver protocol class should be pluggable as well.

You had me at "pluggable."

Initial commit made in newclient branch. Still very, very inchoate.

Also any coaching on how to mark bugs etc (maybe a link to relevant documents) I would really appreciate. I've used lp off and on and now I think I like it better than github/bitbucket.

On Mon, Feb 6, 2012 at 11:04 AM, Drew Smathers
<email address hidden> wrote:
>
> Also any coaching on how to mark bugs etc (maybe a link to relevant
> documents) I would really appreciate.

I dont think there's anything in place for that yet...

But I've just created a new blueprint for general EC2 improvements:
  https://blueprints.launchpad.net/txaws/+spec/ec2-improvements

And I've linked your bug to it. Feel free to file others as you see
them, and link them to that blueprint.

> I've used lp off and on and now I
> think I like it better than github/bitbucket.

I actually *really* like LP for managing tasks and bugs (and building
PPAs, etc.) but I have to say, that after using git for about 6 months
now, I have a hard time going back to bzr. git is sooo much faster and
I really like having all my branches maintained in a single
repo/working directory. It really simplifies my coding workflow.

But compared to svn or cvs pain, the bzr-git gap is a mere speck --
quite easy to live with.

On Mon, Feb 6, 2012 at 11:04 AM, Drew Smathers
<email address hidden> wrote:
>
> Initial commit made in newclient branch. Still very, very inchoate.

Just branched it and took a look at the diff -- looks like you're off
to a good start!

Awesome. Thanks for the feedback.

This is kind of off-topic for this ticket, but bzr has supported a colocated branch workflow (single working tree housing a repository and multiple branches) for a while now.

Tristan, yes off topic ;-) But ... that sounds awesome. And I'll look into this later. Any docs you'd recommend reading on the subject?

Look at the bzr-colo plugin and associated docs.

Ok ... so I got a little ahead of myself on the newclient branch and went into extending APIs in s3 to support plugging in a response body receiver factor and, per API call to the client, an IBodyProducer. I think those aspects are better reflected in a separate ticket ... really just this one I think: #767205

I will probably also create a new branch for these lower-level changes (ditch newclient) and cherry pick or merge relevant changes into that new branch ... if I can learn how to do that with bzr (I'm mostly fluent only in git right now).
.

One question: How much do we care about backwards compat?? There are some public members (client, factory ...) on BaseQuery object that could be used (are being used internally in txaws in various places not covered by tests apparently), so this causes me some concern. Trying to retain these members with some compatibility seems horribly messy to me, so I'm conflicted as to how to approach this. So, any suggestions?

Also just to note some functionality that broke which needs to be resolved in addition to other things:

$ ./bin/txaws-discover --key $AWS_ACCESS_KEY_ID --secret $AWS_SECRET_ACCESS_KEY --action DescribeRegions --endpoint ec2.ap-southeast-1.amazonaws.com
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/web/_newclient.py", line 1034, in _bodyDataFinished_CONNECTED
    self._bodyProtocol.connectionLost(reason)
  File "./txaws/client/base.py", line 116, in connectionLost
    d.callback(self._buffer.getvalue())
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 368, in callback
    self._startRunCallbacks(result)
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "./txaws/client/discover/command.py", line 72, in write_error
    print >> self.output, "URL: %s" % query.client.url
exceptions.AttributeError: 'Query' object has no attribute 'client'

N.B. I've made an additional change since proposing for merge. The StringIOBodyReceiver I think is better generalized as StreamingBodyReceiver which can be useful for streaming to file. Example:

fd = open('data.bin', 'w')
query = BaseQuery(..., receiver_factory=lambda: StreamingBodyReceiver(fd, readback=False)