Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move pynsq #88

Merged
merged 2 commits into from
Oct 31, 2012
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ data format (messages can be JSON, [MsgPack][msgpack], [Protocol Buffers][go-pro
else). Official Go and Python libraries are available out of the box and, if you're interested in
building your own client, there's a [protocol spec][protocol] (see [client libraries](#client)).

We publish [binary releases][binary] for linux and darwin.
The latest stable release is **[0.2.15][latest_tag]**. We publish [binary releases][binary] for
linux and darwin.

NOTE: master is our *development* branch and may *not* be stable at all times.

[![Build Status](https://secure.travis-ci.org/bitly/nsq.png)](http://travis-ci.org/bitly/nsq)

Expand All @@ -32,11 +35,11 @@ doc][design] or [blog post][nsq_post].

## <a name="client"></a>Client Libraries

* [Go (official)][nsq]
* [Python (official)][pynsq]
* [Node.js][node_lib]
* [PHP][php_lib]
* [Ruby][ruby_lib]
* [nsq][nsq] Go (official)
* [pynsq][pynsq] Python (official) [pypi][pynsq_pypi]
* [nodensq][node_lib] Node.js [npm][nodensq_npm]
* [nsqphp][php_lib] PHP
* [ruby_nsq][ruby_lib] Ruby [rubygems][ruby_nsq_rubygems]

## Additional Documentation

Expand All @@ -46,7 +49,6 @@ doc][design] or [blog post][nsq_post].
* [nsqlookupd][nsqlookupd] is the daemon that manages topology information
* [nsqadmin][nsqadmin] is the web UI to view message statistics and perform administrative tasks
* [nsq][nsq] is a go package for writing `nsqd` clients
* [pynsq][pynsq] is a python module for writing `nsqd` clients

There is also a [protocol spec][protocol].

Expand Down Expand Up @@ -116,7 +118,7 @@ NSQ was designed and developed by Matt Reiferson ([@imsnakes][snakes_twitter]) a
[nsqlookupd]: https://github.com/bitly/nsq/tree/master/nsqlookupd
[nsqadmin]: https://github.com/bitly/nsq/tree/master/nsqadmin
[nsq]: https://github.com/bitly/nsq/tree/master/nsq
[pynsq]: https://github.com/bitly/nsq/tree/master/pynsq
[pynsq]: https://github.com/bitly/pynsq
[nsq_post]: http://word.bitly.com/post/33232969144/nsq
[binary]: https://github.com/bitly/nsq/downloads
[snakes_twitter]: https://twitter.com/imsnakes
Expand All @@ -138,4 +140,7 @@ NSQ was designed and developed by Matt Reiferson ([@imsnakes][snakes_twitter]) a
[spof]: https://github.com/bitly/nsq/blob/master/docs/design.md#spof
[message_guarantee]: https://github.com/bitly/nsq/blob/master/docs/design.md#delivery
[design]: https://github.com/bitly/nsq/blob/master/docs/design.md

[latest_tag]: https://github.com/bitly/nsq/tree/v0.2.15
[pynsq_pypi]: http://pypi.python.org/pypi/pynsq
[nodensq_npm]: https://npmjs.org/package/nsq
[ruby_nsq_rubygems]: http://rubygems.org/gems/ruby_nsq
95 changes: 1 addition & 94 deletions pynsq/README.md
Original file line number Diff line number Diff line change
@@ -1,94 +1 @@
## pynsq

`pynsq` is a Python NSQ client library.

It provides a high-level reader library for building consumers and two low-level modules for both
sync and async communication over the NSQ protocol (if you wanted to write your own high-level
functionality).

The async module is built on top of the [Tornado IOLoop][tornado] and as such requires `tornado` be
installed:

`$ pip install tornado`

### Reader

Reader provides high-level functionality for building robust NSQ consumers in Python on top of the
async module.

Multiple reader instances can be instantiated in a single process (to consume from multiple
topics/channels at once). Each specifying a set of tasks that will be called for each message over
that channel. Tasks are defined as a dictionary of string names -> callables passed as
`all_tasks` during instantiation.

`preprocess_method` defines an optional callable that can alter the message data before other task
functions are called.

`validate_method` defines an optional callable that returns a boolean as to weather or not this
message should be processed.

`async` determines whether handlers will do asynchronous processing. If set to True, handlers must
accept a keyword argument called `finisher` that will be a callable used to signal message
completion (with a boolean argument indicating success).

The library handles backoff as well as maintaining a sufficient RDY count based on the # of
producers and your configured `max_in_flight`.

Here is an example that demonstrates synchronous message processing:

```python
import nsq

def task1(message):
print message
return True

def task2(message):
print message
return True

all_tasks = {"task1": task1, "task2": task2}
r = nsq.Reader(all_tasks, lookupd_http_addresses=['http://127.0.0.1:4161'],
topic="nsq_reader", channel="asdf")
nsq.run()
```

And async:

```python
"""
This is a simple example of async processing with nsq.Reader.

It will print "deferring processing" twice, and then print
the last 3 messages that it received.

Note in particular that we pass the `async=True` argument to Reader(),
and also that we cache a different finisher callable with
each message, to be called when we have successfully finished
processing it.
"""
import nsq

buf = []

def process_message(message, finisher):
global buf
# cache both the message and the finisher callable for later processing
buf.append((message, finisher))
if len(buf) >= 3:
print '****'
for msg, finish_fxn in buf:
print msg
finish_fxn(True) # use finish_fxn to tell NSQ of success
print '****'
buf = []
else:
print 'deferring processing'

all_tasks = {"task1": process_message}
r = nsq.Reader(all_tasks, lookupd_http_addresses=['http://127.0.0.1:4161'],
topic="nsq_reader", channel="async", async=True)
nsq.run()
```

[tornado]: https://github.com/facebook/tornado
**pynsq** has moved to it's [own repository][https://github.com/bitly/pynsq]
68 changes: 0 additions & 68 deletions pynsq/nsq/BackoffTimer.py

This file was deleted.

Loading