Table Of Contents

Introduction

gevent is a Python networking library that uses greenlet to provide a synchronous API on top of libevent event loop.

Features include:

  • Fast event loop based on libevent (epoll on Linux, kqueue on FreeBSD).
  • Lightweight execution units based on greenlet.
  • API that re-uses concepts from the Python standard library (e.g. Event, Queue).
  • Cooperative socket and ssl modules.
  • Ability to use standard library and 3rd party modules written for standard blocking sockets (gevent.monkey).
  • DNS queries performed through libevent-dns.
  • Fast WSGI server based on libevent-http.

Installation

gevent runs on Python 2.4 and newer and requires

For ssl to work on Python older than 2.6, ssl package is required.

Example

The following example shows how to run tasks concurrently.

>>> import gevent
>>> from gevent import socket
>>> urls = ['www.google.com', 'www.example.com', 'www.python.org']
>>> jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]
>>> gevent.joinall(jobs, timeout=2)
>>> [job.value for job in jobs]
['74.125.79.106', '208.77.188.166', '82.94.164.162']

After the jobs have been spawned, gevent.joinall() waits for them to complete, no longer than 2 seconds though. The results are then collected by checking gevent.Greenlet.value property. The gevent.socket.gethostbyname() function has the same interface as the standard socket.gethostbyname() but it does not block the whole interpreter and thus lets the other greenlets to proceed with their requests as well.

Monkey patching

The example above used gevent.socket for socket operations. If the standard socket module was used it would took it 3 times longer to complete because the DNS requests would be sequential. Using the standard socket module inside greenlets makes gevent rather pointless, so what about module and packages that are built on top of socket?

That’s what monkey patching for. The functions in gevent.monkey carefully replace functions and classes in the standard socket module with their cooperative counterparts. That way even the modules that are unaware of gevent can benefit from running in multi-greenlet environment.

>>> from gevent import monkey; monkey.patch_socket()
>>> import urllib2 # it's usable from multiple greenlets now

See examples/concurrent_download.py

Event loop

Unlike other network libraries and similar to eventlet, gevent starts the event loop implicitly in a dedicated greenlet. There’s no reactor that you must run() or dispatch() function to call. When a function from gevent API wants to block, it obtains the Hub instance - a greenlet that runs the event loop - and switches to it. If there’s no Hub instance yet, one is created on the fly.

The event loop provided by libevent uses the fastest polling mechanism available on the system by default. It is possible to command libevent not to use a particular polling mechanism by setting EVENT_NOXXX` environment variable where XXX is the event loop you want to disable. For example, on Linux setting EVENT_NOEPOLL=1 would avoid the default epoll mechanism and use poll.

Libevent API is available under gevent.core module. Note, that the callbacks supplied to libevent API are run in the Hub greenlet and thus cannot use synchronous gevent API. It is possible to use asynchronous API there, like spawn() and Event.set().

Cooperative multitasking

The greenlets all run in the same OS thread and scheduled cooperatively. This means that until a particular greenlet gives up control, by calling a blocking function that will switch to the Hub, other greenlets won’t get a chance to run. It is typically not an issue for an I/O bound app, but one should be aware of this when doing something CPU intensive or calling blocking I/O functions that bypass libevent event loop.

Synchronizing access to the objects shared across the greenlets is unnecessary in most cases, thus Lock and Semaphore classes although present aren’t used as often. Other abstractions from threading and multiprocessing remain useful in the cooperative world:

  • Event allows to wake up a number of greenlets that are calling Event.wait() method.
  • AsyncResult is similar to Event but allows passing a value or an exception to the waiters.
  • Queue and JoinableQueue.

Lightweight pseudothreads

The greenlets are spawned by creating a Greenlet instance and calling its start method. (The spawn() function is a shortcut that does exactly that). The start method schedules an event that will switch to the greenlet created, as soon as the current greenlet gives up control. If there is more than one active event, they will be executed one by one, in an undefined order.

If there was an error during execution it won’t escape greenlet’s boundaries. An unhandled error results in a stacktrace being printed complemented by failed function signature and arguments:

>>> gevent.spawn(lambda : 1/0)
>>> gevent.sleep(1)
Traceback (most recent call last):
 ...
ZeroDivisionError: integer division or modulo by zero
<Greenlet at 0x7f2ec3a4e490: <function <lambda...>> failed with ZeroDivisionError

The traceback is asynchronously printed to sys.stderr when the greenlet dies.

Greenlet instances has a number of useful methods:

  • join – waits until the greenlet exits;
  • kill – interrupts greenlet’s execution;
  • get – returns the value returned by greenlet or re-raised the exception that killed it.

It is possible to customize the string printed after the traceback by subclassing Greenlet class and redefining its __str__ method.

To subclass a Greenlet, override its _run() method and call Greenlet.__init__(self) in __init__:

class MyNoopGreenlet(Greenlet):

    def __init__(self, seconds):
        Greenlet.__init__(self)
        self.seconds = seconds

    def _run(self):
        gevent.sleep(self.seconds)

    def __str__(self):
        return 'MyNoopGreenlet(%s)' % self.seconds

Greenlets can be killed asynchronously. Killing will resume the sleeping greenlet, but instead of continuing execution, a GreenletExit will be raised.

>>> g = MyNoopGreenlet(4)
>>> g.start()
>>> g.kill()
>>> g.dead
True

The GreenletExit exception and its subclasses are handled differently then other exceptions. Raising GreenletExit is not considered an exceptional situation, so the traceback is not printed. The GreenletExit is returned by get as if it was returned by the greenlet, not raised.

The kill method can accept an exception to raise:

>>> g = MyNoopGreenlet.spawn(5) # spawn() creates a Greenlet and starts it
>>> g.kill(Exception("A time to kill"))
Traceback (most recent call last):
 ...
Exception: A time to kill
MyNoopGreenlet(5) failed with Exception

The kill can also accept a timeout argument specifying the number of seconds to wait for the greenlet to exit. Note, that kill cannot guarantee that the target greenlet will not ignore the exception and thus it’s a good idea always to pass a timeout to kill.

Timeouts

Many functions in the gevent API are synchronous, blocking the current greenlet until the operation is done. For example, kill waits until the target greenlet is dead before returning [1]. Many of those functions can be made asynchronous by passing block=False argument.

Furthermore, many of the synchronous functions accept timeout argument, which specifies a limit on how long the function could block (examples: Event.wait(), Greenlet.join(), Greenlet.kill(), AsyncResult.get() and many more).

The socket and SSLObject instances can also have a timeout, set by settimeout method.

When these are not enough, the Timeout class can be used to add timeouts to arbitrary sections of (yielding) code.

Futher reading

To limit concurrency, use Pool class (see example: dns_mass_resolve.py).

Gevent comes with TCP/SSL/HTTP/WSGI servers. See Implementing servers.

Footnotes

[1]This was not the case before 0.13.0, kill method in 0.12.2 and older was asynchronous by default.

API reference

gevent – basic utilities

The most common functions and classes are available in the gevent top level package.

Greenlet objects

Greenlet is a light-weight cooperatively-scheduled execution unit.

To start a new greenlet, pass the target function and its arguments to Greenlet constructor and call start():

>>> g = Greenlet(myfunction, 'arg1', 'arg2', kwarg1=1)
>>> g.start()

or use classmethod spawn() which is a shortcut that does the same:

>>> g = Greenlet.spawn(myfunction, 'arg1', 'arg2', kwarg1=1)

To subclass a Greenlet, override its _run() method and call Greenlet.__init__(self) in __init__(): It also a good idea to override __str__(): if _run() raises an exception, its string representation will be printed after the traceback it generated.

class gevent.Greenlet
Greenlet.value

Holds the value returned by the function if the greenlet has finished successfully. Otherwise None.

Greenlet.exception

Holds the exception instance raised by the function if the greenlet has finished with an error. Otherwise None.

Greenlet.ready()

Return true if and only if the greenlet has finished execution.

Greenlet.successful()

Return true if and only if the greenlet has finished execution successfully, that is, without raising an error.

Greenlet.start()

Schedule the greenlet to run in this loop iteration

Greenlet.start_later(seconds)

Schedule the greenlet to run in the future loop iteration seconds later

Greenlet.join(timeout=None)

Wait until the greenlet finishes or timeout expires. Return None regardless.

Greenlet.get(block=True, timeout=None)

Return the result the greenlet has returned or re-raise the exception it has raised.

If block is False, raise gevent.Timeout if the greenlet is still alive. If block is True, unschedule the current greenlet until the result is available or the timeout expires. In the latter case, gevent.Timeout is raised.

Greenlet.kill(exception=GreenletExit, block=False, timeout=None)

Raise the exception in the greenlet.

If block is True (the default), wait until the greenlet dies or the optional timeout expires. If block is False, the current greenlet is not unscheduled.

The function always returns None and never raises an errir.

Changed in version 0.13.0: block is now True by default.

Link greenlet’s completion to callable or another greenlet.

If receiver is a callable then it will be called with this instance as an argument once this greenlet’s dead. A callable is called in its own greenlet.

If receiver is a greenlet then an LinkedExited exception will be raised in it once this greenlet’s dead.

If receiver is None, link to the current greenlet.

Always asynchronous, unless receiver is a current greenlet and the result is ready. If this greenlet is already dead, then notification will performed in this loop iteration as soon as this greenlet switches to the hub.

Like link() but receiver is only notified when the greenlet has completed successfully

Like link() but receiver is only notified when the greenlet dies because of unhandled exception

Remove the receiver set by link() or rawlink()

Being a greenlet subclass, Greenlet also has switch() and throw() methods. However, these should not be used at the application level. Prefer higher-level safe classes, like Event and Queue, instead.

exception gevent.GreenletExit

A special exception that kills the greenlet silently.

When a greenlet raises GreenletExit or a subclass, the traceback is not printed and the greenlet is considered successful. The exception instance is available under value property as if it was returned by the greenlet, not raised.

Spawn helpers

gevent.spawn(function, *args, **kwargs)

Create a new Greenlet object and schedule it to run function(*args, **kwargs). This is an alias for Greenlet.spawn().

gevent.spawn_later(seconds, function, *args, **kwargs)

Create a new Greenlet object and schedule it to run function(*args, **kwargs) in the future loop iteration seconds later. This is an alias for Greenlet.spawn_later().

gevent.spawn_raw(function, *args, **kwargs)

Create a new greenlet object and schedule it to run function(*args, **kwargs). As this returns a raw greenlet, it does not have all the useful methods that gevent.Greenlet has and should only be used as an optimization.

This are the shortcuts for:

g = spawn(function, *args, **kwargs)
g.link() # or g.link_value() or g.link_exception()

As Greenlet.link() without argument links to the current greenlet, a gevent.greenlet.LinkedExited exception will be raised if the newly spawned greenlet exits. It is not meant as a way of inter-greenlet communication but more of a way to assert that a background greenlet is running at least as long as the current greenlet.

See Greenlet.link(), Greenlet.link_value() and Greenlet.link_exception() for details.

Useful general functions

gevent.getcurrent()
gevent.sleep(seconds=0)

Put the current greenlet to sleep for at least seconds.

seconds may be specified as an integer, or a float if fractional seconds are desired. Calling sleep with seconds of 0 is the canonical way of expressing a cooperative yield.

gevent.kill(greenlet, exception=GreenletExit)

Kill greenlet asynchronously. The current greenlet is not unscheduled.

Note, that gevent.Greenlet.kill() method does the same and more. However, MAIN greenlet - the one that exists initially - does not have kill() method so you have to use this function.

gevent.killall(greenlets, exception=GreenletExit, block=False, timeout=None)
gevent.joinall(greenlets, timeout=None, raise_error=False)
gevent.signal(signalnum, handler, *args, **kwargs)
gevent.fork()
gevent.shutdown()

Cancel our CTRL-C handler and wait for core.dispatch() to return.

gevent.reinit()

Wrapper for event_reinit().

Timeouts

class gevent.Timeout(seconds=None, exception=None)

Raise exception in the current greenlet after given time period:

timeout = Timeout(seconds, exception)
timeout.start()
try:
    ...  # exception will be raised here, after *seconds* passed since start() call
finally:
    timeout.cancel()

When exception is omitted or None, the Timeout instance itself is raised:

>>> Timeout(0.1).start()
>>> gevent.sleep(0.2)
Traceback (most recent call last):
 ...
Timeout: 0.1 seconds

For Python 2.5 and newer with statement can be used:

with Timeout(seconds, exception) as timeout:
    pass  # ... code block ...

This is equivalent to try/finally block above with one additional feature: if exception is False, the timeout is still raised, but context manager suppresses it, so the code outside the with-block won’t see it.

This is handy for adding a timeout to the functions that don’t support timeout parameter themselves:

data = None
with Timeout(5, False):
    data = mysock.makefile().readline()
if data is None:
    ...  # 5 seconds passed without reading a line
else:
    ...  # a line was read within 5 seconds

Note that, if readline() above catches and doesn’t re-raise BaseException (for example, with except:), then your timeout is screwed.

When catching timeouts, keep in mind that the one you catch maybe not the one you have set; if you going to silent a timeout, always check that it’s the one you need:

timeout = Timeout(1)
timeout.start()
try:
    ...
except Timeout, t:
    if t is not timeout:
        raise # not my timeout
pending

Return True if the timeout is scheduled to be raised.

start()

Schedule the timeout.

classmethod start_new(timeout=None, exception=None)

Create a started Timeout.

This is a shortcut, the exact action depends on timeout‘s type:

  • If timeout is a Timeout, then call its start() method.
  • Otherwise, create a new Timeout instance, passing (timeout, exception) as arguments, then call its start() method.

Returns the Timeout instance.

cancel()

If the timeout is pending, cancel it. Otherwise, do nothing.

gevent.with_timeout(seconds, function, *args, **kwds)

Wrap a call to function with a timeout; if the called function fails to return before the timeout, cancel it and return a flag value, provided by timeout_value keyword argument.

If timeout expires but timeout_value is not provided, raise Timeout.

Keyword argument timeout_value is not passed to function.

Networking interfaces

Synchronization primitives

gevent.event – Notifications of multiple listeners

class gevent.event.Event

A synchronization primitive that allows one greenlet to wake up one or more others. It has the same interface as threading.Event but works across greenlets.

An event object manages an internal flag that can be set to true with the set() method and reset to false with the clear() method. The wait() method blocks until the flag is true.

is_set()
isSet()

Return true if and only if the internal flag is true.

set()

Set the internal flag to true. All greenlets waiting for it to become true are awakened. Greenlets that call wait() once the flag is true will not block at all.

clear()

Reset the internal flag to false. Subsequently, threads calling wait() will block until set() is called to set the internal flag to true again.

wait(timeout=None)

Block until the internal flag is true. If the internal flag is true on entry, return immediately. Otherwise, block until another thread calls set() to set the flag to true, or until the optional timeout occurs.

When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof).

Return the value of the internal flag (True or False).

Register a callback to call when the internal flag is set to true.

callback will be called in the Hub, so it must not use blocking gevent API. callback will be passed one argument: this instance.

Remove the callback set by rawlink()

class gevent.event.AsyncResult

A one-time event that stores a value or an exception.

Like Event it wakes up all the waiters when set() or set_exception() method is called. Waiters may receive the passed value or exception by calling get() method instead of wait(). An AsyncResult instance cannot be reset.

To pass a value call set(). Calls to get() (those that currently blocking as well as those made in the future) will return the value:

>>> result = AsyncResult()
>>> result.set(100)
>>> result.get()
100

To pass an exception call set_exception(). This will cause get() to raise that exception:

>>> result = AsyncResult()
>>> result.set_exception(RuntimeError('failure'))
>>> result.get()
Traceback (most recent call last):
 ...
RuntimeError: failure

AsyncResult implements __call__() and thus can be used as link() target:

>>> result = AsyncResult()
>>> gevent.spawn(lambda : 1/0).link(result)
>>> result.get()
Traceback (most recent call last):
 ...
ZeroDivisionError: integer division or modulo by zero
value

Holds the value passed to set() if set() was called. Otherwise None.

exception

Holds the exception instance passed to set_exception() if set_exception() was called. Otherwise None.

ready()

Return true if and only if it holds a value or an exception

successful()

Return true if and only if it is ready and holds a value

set(value=None)

Store the value. Wake up the waiters.

All greenlets blocking on get() or wait() are woken up. Sequential calls to wait() and get() will not block at all.

set_exception(exception)

Store the exception. Wake up the waiters.

All greenlets blocking on get() or wait() are woken up. Sequential calls to wait() and get() will not block at all.

get(block=True, timeout=None)

Return the stored value or raise the exception.

If this instance already holds a value / an exception, return / raise it immediatelly. Otherwise, block until another greenlet calls set() or set_exception() or until the optional timeout occurs.

When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof).

get_nowait()

Return the value or raise the exception without blocking.

If nothing is available, raise gevent.Timeout immediatelly.

wait(timeout=None)

Block until the instance is ready.

If this instance already holds a value / an exception, return immediatelly. Otherwise, block until another thread calls set() or set_exception() or until the optional timeout occurs.

When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof).

Return value.

Register a callback to call when a value or an exception is set.

callback will be called in the Hub, so it must not use blocking gevent API. callback will be passed one argument: this instance.

Remove the callback set by rawlink()

gevent.queue – Synchronized queues

The gevent.queue module implements multi-producer, multi-consumer queues that work across greenlets, with the API similar to the classes found in the standard Queue and multiprocessing modules.

A major difference is that queues in this module operate as channels when initialized with maxsize of zero. In such case, both Queue.empty() and Queue.full() return True and Queue.put() always blocks until a call to Queue.get() retrieves the item.

Another interesting difference is that Queue.qsize(), Queue.empty(), and Queue.full() can be used as indicators of whether the subsequent Queue.get() or Queue.put() will not block.

class gevent.queue.Queue(maxsize=None)

Create a queue object with a given maximum size.

If maxsize is less than zero or None, the queue size is infinite.

Queue(0) is a channel, that is, its put() method always blocks until the item is delivered. (This is unlike the standard Queue, where 0 means infinite size).

qsize()

Return the size of the queue.

empty()

Return True if the queue is empty, False otherwise.

full()

Return True if the queue is full, False otherwise.

Queue(None) is never full.

put(item, block=True, timeout=None)

Put an item into the queue.

If optional arg block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).

put_nowait(item)

Put an item into the queue without blocking.

Only enqueue the item if a free slot is immediately available. Otherwise raise the Full exception.

get(block=True, timeout=None)

Remove and return an item from the queue.

If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).

get_nowait()

Remove and return an item from the queue without blocking.

Only get an item if one is immediately available. Otherwise raise the Empty exception.

class gevent.queue.PriorityQueue(maxsize=None)

A subclass of Queue that retrieves entries in priority order (lowest first).

Entries are typically tuples of the form: (priority number, data).

class gevent.queue.LifoQueue(maxsize=None)

A subclass of Queue that retrieves most recently added entries first.

class gevent.queue.JoinableQueue(maxsize=None)

A subclass of Queue that additionally has task_done() and join() methods.

task_done()

Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put into the queue).

Raises a ValueError if called more times than there were items placed in the queue.

join()

Block until all items in the queue have been gotten and processed.

The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate that the item was retrieved and all work on it is complete. When the count of unfinished tasks drops to zero, join() unblocks.

exception gevent.queue.Full

An alias for Queue.Full

exception gevent.queue.Empty

An alias for Queue.Empty

Example of how to wait for enqueued tasks to be completed:

def worker():
    while True:
        item = q.get()
        try:
            do_work(item)
        finally:
            q.task_done()

q = JoinableQueue()
for i in range(num_worker_threads):
     gevent.spawn(worker)

for item in source():
    q.put(item)

q.join()  # block until all tasks are done

Implementing servers

There are a few classes to simplify server implementation with gevent. They all share the similar interface:

def handle(socket, address):
     print 'new connection!'

server = StreamServer(('127.0.0.1', 1234), handle) # creates a new server
server.start() # start accepting new connections

At this point, any new connection accepted on 127.0.0.1:1234 will result in a new Greenlet spawned using handle function. To stop a server use stop() method.

In case of a WSGIServer, handle must be a WSGI application callable.

It is possible to limit the maximum number of concurrent connections, by passing a Pool instance:

pool = Pool(10000) # do not accept more than 10000 connections
server = StreamServer(('127.0.0.1', 1234), handle, spawn=pool)
server.serve_forever()

The server_forever() method calls start() and then waits until interrupted or until the server is stopped.

The difference between wsgi.WSGIServer and pywsgi.WSGIServer is that the first one is very fast as it uses libevent’s http server implementation but it shares the issues that libevent-http has. In particular:

The pywsgi.WSGIServer does not have these limitations. In addition, gunicorn is a stand-alone server that supports gevent. Gunicorn has its own HTTP parser but can also use gevent.wsgi module.

More examples are available in the code repository:

gevent.core - Low-level wrappers around libevent

This module provides a mechanism to execute a function when a specific event on a file handle, file descriptor, or signal occurs, or after a given time has passed. It also provides wrappers around structures and functions from libevent-dns and libevent-http.

This module does not work with the greenlets. A callback passed to a method from this module will be executed in the event loop, which is running in the Hub greenlet. Therefore it must not use any synchronous gevent API, that is, the functions that switch to the Hub. It’s OK to call asynchronous stuff like gevent.spawn(), Event.set <gevent.event.Event.set() or Queue.put_nowait.

The code is based on pyevent.

events

class gevent.core.event(evtype, handle, callback[, arg])

Create a new event object with a user callback.

  • evtype – bitmask of EV_READ or EV_WRITE, or EV_SIGNAL
  • handle – a file handle, descriptor, or socket for EV_READ or EV_WRITE; a signal number for EV_SIGNAL
  • callback – user callback with (event, evtype) prototype
  • arg – optional object, which will be made available as arg property.
add()

Add event to be executed after an optional timeout - number of seconds after which the event will be executed.

arg
callback
cancel()

Remove event from the event queue.

events
events_str
fd
flags
flags_str
pending

Return True if the event is still scheduled to run.

class gevent.core.read_event

Create a new scheduled event with evtype=EV_READ

class gevent.core.write_event

Create a new scheduled event with evtype=EV_WRITE

class gevent.core.timer

Create a new scheduled timer

class gevent.core.signal

Create a new persistent signal event

class gevent.core.active_event

An event that is scheduled to run in the current loop iteration

add()

event loop

gevent.core.init()

Initialize event queue.

gevent.core.dispatch()

Dispatch all events on the event queue. Returns 0 on success, and 1 if no events are registered. May raise IOError.

gevent.core.loop()

Dispatch all pending events on queue in a single pass. Returns 0 on success, and 1 if no events are registered. May raise IOError.

gevent.core.get_version()

Wrapper for event_get_version()

gevent.core.get_method()

Wrapper for event_get_method()

gevent.core.get_header_version()

Return _EVENT_VERSION

evdns

gevent.core.dns_init()

Initialize async DNS resolver.

gevent.core.dns_shutdown()

Shutdown the async DNS resolver and terminate all active requests.

gevent.core.dns_resolve_ipv4()

Lookup an A record for a given name.

  • name – DNS hostname
  • flags – either 0 or DNS_QUERY_NO_SEARCH
  • callback – callback with (result, type, ttl, addrs) prototype
gevent.core.dns_resolve_ipv6()

Lookup an AAAA record for a given name.

  • name – DNS hostname
  • flags – either 0 or DNS_QUERY_NO_SEARCH
  • callback – callback with (result, type, ttl, addrs) prototype
gevent.core.dns_resolve_reverse()

Lookup a PTR record for a given IPv4 address.

  • packed_ip – IPv4 address (as 4-byte binary string)
  • flags – either 0 or DNS_QUERY_NO_SEARCH
  • callback – callback with (result, type, ttl, addrs) prototype
gevent.core.dns_resolve_reverse_ipv6()

Lookup a PTR record for a given IPv6 address.

  • packed_ip – IPv6 address (as 16-byte binary string)
  • flags – either 0 or DNS_QUERY_NO_SEARCH
  • callback – callback with (result, type, ttl, addrs) prototype

evbuffer

class gevent.core.buffer

file-like wrapper for libevent’s evbuffer structure.

Note, that the wrapper does not own the structure, libevent does.

detach()
next
read()

Drain the first size bytes from the buffer (or what’s left if there are less than size bytes).

If size is negative, drain the whole buffer.

readline()
readlines()

evhttp

class gevent.core.http_request

Wrapper around libevent’s evhttp_request structure.

add_input_header()
add_output_header()
chunked
clear_input_headers()
clear_output_headers()
connection
default_response_headers
detach()
find_input_header()
find_output_header()
get_input_headers()
input_buffer
kind
major
minor
output_buffer
remote
remote_host
remote_port
remove_input_header()

Return True if header was found and removed

remove_output_header()

Return True if header was found and removed

response
response_code
response_code_line
send_error()
send_reply()
send_reply_chunk()
send_reply_end()
send_reply_start()
type
typestr
uri
version
class gevent.core.http_connection
peer
class gevent.core.http
accept()
bind()
default_response_headers
handle

Changelog

Release 0.13.1 (Sep 23, 2010)

Release highlights:

  • Fixed monkey to patch socket.create_connection.
  • Updated gevent.ssl module to fully match the functionality of ssl on Python 2.7.
  • Fixed Group.join() to handle raise_error=True properly, it used to raise TypeError (issue #36). Thanks to by David Hain.
  • Fixed gevent.wsgi and gevent.pywsgi to join multiple Cookie headers (issue #40).
  • Fixed select to recognize long arguments in addition to int.
  • Fixed Semaphore.acquire() to return False when timeout expires instead of raising AssertionError (issue #39). Patch by Teh Ekik.
  • Fixed JoinableQueue.join() to return immediatelly if queue is already empty (issue #45). Patch by Dmitry Chechik.
  • Deprecated gevent.sslold module.

gevent.socket module:

  • Overrode socket.shutdown() method to interrupt read/write operations on socket.
  • Fixed possible NameError in socket.connect_ex() method. Patch by Alexey Borzenkov.
  • Fixed socket leak in create_connection() function.
  • Made gevent.socket import all public items from stdlib socket that do not do I/O.

gevent.ssl module:

  • Imported a number of patches from stdlib by Antoine Pitrou:
    • Calling makefile() method on an SSL object would prevent the underlying socket from being closed until all objects get truely destroyed (Python issue #5238).
    • SSL handshake would ignore the socket timeout and block indefinitely if the other end didn’t respond (Python issue #5103).
    • When calling getpeername() in SSLSocket.__init__, only silence exceptions caused by the “socket not connected” condition.
  • Added support for ciphers argument.
  • Updated SSLSocket.send and SSLSocket.recv methods to match the behavior of stdlib ssl better.
  • Fixed ssl.SSLObject to delete events used by other greenlets when closing the instance (issue #34).

Miscellaneous:

  • Made BaseServer accept long values as pool argument in addition to int.
  • Made http._requests attribute public.
  • Updated webchat example to use file on disk rather than in-memory sqlite database to avoid OperationalError.
  • Fixed webproxy.py example to be runnable under external WSGI server.
  • Fixed bogus failure in test__exc_info.py.
  • Added new test to check PEP8 conformance: xtest_pep8.py.
  • Fixed BackdoorServer close the connection on SystemExit and simplified the code.
  • Made Pool raise ValueError when initialized with size=0.
  • Updated setup.py --libevent to configure and make libevent if it’s built already.
  • Updated setup.py to use setuptools if present and add dependency on greenlet.
  • Fixed doc/mysphinxext.py to work with Sphinx 1. Thanks by Örjan Persson.

Release 0.13.0 (Jul 14, 2010)

Release highlights:

  • Added gevent.server module with StreamServer class for easy implementing of TCP and SSL servers.
  • Added gevent.baseserver module with BaseServer class.
  • Added new implementation of gevent.pywsgi based on gevent.server. Contributed by Ralf Schmitt.
  • Added gevent.local module. Fixed issue #24. Thanks to Ted Suzman.
  • Fixed a number of bugs in gevent.wsgi module.
  • Fixed issue #26: closing a socket now interrupts all pending read/write operations on it.
  • Implemented workaround that prevents greenlets from leaking exc_info.
  • Fixed socket.sendall() to use buffer object to prevent string copies.
  • Made the interfaces of gevent.wsgi and gevent.pywsgi much more similar to each other.
  • Fixed compilation on Windows with libevent-2.
  • Improved Windows compatibility. Fixed issue #30. Thanks to Luigi Pugnetti.
  • Fixed compatibility with Python 2.7.

Backward-incompatible changes:

  • Blocking is now the default behaviour for the Greenlet.kill() method and other kill* methods.
  • Changed the inteface of http.HTTPServer to match the interface of other servers.
  • Changed Pool‘s spawn() method to block until there’s a free slot.
  • Removed deprecated backdoor.backdoor_server() function.
  • Removed deprecated functions in socket module:
    • socket_bind_and_listen()
    • set_reuse_addr()
    • connect_tcp()
    • tcp_server()
  • Removed deprecated socket.fd property.
  • Deprecated use of negative numbers to indicate infinite timeout in core.event.add() and socket.wait_read() and similar. Use None from now on, which is compatible with the previous versions.
  • Derived backdoor.BackdoorServer from StreamServer rather than from Greenlet. This adds lots of new features and removes a few old ones.
  • Removed non-standard balance property from Semaphore.
  • Removed start(), set_cb() and set_gencb() from core.http.
  • Removed set_closecb() from core.http_connection. It is now used internally to detach the requests of the closed connections.
  • Deprecated rawgreenlet module.
  • Deprecated util.lazy_property().
  • Renamed GreenletSet to Group. The old name is currently available as an alias.

gevent.socket module:

  • Fixed issues #26 and #34: closing the socket while reading/writing/connecting is now safe. Thanks to Cyril Bay.
  • Imported getfqdn() from socket module.
  • The module now uses sys.platform to detect Windows rather than platform module.
  • Fixed issue #27: getaddrinfo() used to handle the case when socktype or proto were equal to 0. Thanks to Randall Leeds.

gevent.coros module:

  • Added RLock class.
  • Added DummySemaphore class.
  • Fixed BoundedSemaphore class to behave like threading.BoundedSemaphore behaves.

gevent.event module:

  • Made Event.wait() return internal flag instead of None.
  • Made AsyncResult.wait() return its value instead of None.
  • Added ready() method as an alias for is_set().

gevent.wsgi module:

  • Removed wsgi.buffer_proxy.

gevent.pywsgi module:

  • Rewritten to use server and not to depend on BaseHTTPServer.
  • Changed the interface to match wsgi module. Removed server() function, add Server class, added WSGIServer class.
  • Renamed HttpProtocol to WSGIHandler.
  • Fixed compatibility with webob by allowing an optional argument to readline().

gevent.core module:

  • Fixed reference leaks in event class.
  • Avoid Python name lookups when accessing EV_* constants from Cython code. Patch by Daniele Varrazzo.
  • Added persist argument to read_event, write_event and readwrite_event.
  • Made all of the event loop callbacks clear the exception info before exiting.
  • Added flags_str property to event. It is used by __str__ and __repr__.
  • buffer:
    • Added detach() method.
    • Implemented iterator protocol.
    • Fixed readline() and readlines() methods.
  • http_request:
    • Fixed detach() to detach input and output buffers too.
    • Changed the response to send 500 error upon deallocation, if no response was sent by the user.
    • Made input_buffer and output_buffer store and reuse the buffer object they create.
    • Fixed __str__() and meth:__repr__ to include spaces where needed.
    • http class no longer has set_cb() and set_gencb(). Instead its contructor accepts handle which will be called on each request.

gevent.http and gevent.wsgi modules:

  • Made HTTPServer use "Connection: close" header by default.
  • Class HTTPServer now derives from baseserver.BaseServer. Thus its start() method no longer accepts socket to listen on, it must be passed to the contructor.
  • The spawn argument now accepts a Pool instance. While the pool is full, the server replies with 503 error.
  • The server no longer links to the greenlets it spawns to detect errors. Instead, it relies on http_request which will send 500 reply when deallocated if the user hasn’t send any.

Miscellaneous:

  • Changed gevent.thread to use Greenlet instead of raw greenlets. This means monkey patched thread will become Greenlet too.
  • Added started property to Greenlet.
  • Put common server code in gevent.baseserver module. All servers in gevent package are now derived from BaseServer.
  • Fixed issue #20: sleep() now raises IOError if passed a negative argument.
  • Remove the code related to finding out libevent version from setup.py as macro USE_LIBEVENT_? is no longer needed to build gevent.core.
  • Increased default backlog in all servers (from 5 to 256). Thanks to Nicholas Piël.
  • Fixed doc/conf.py to work in Python older than 2.6. Thanks to Örjan Persson.
  • Silented SystemError raised in backdoor when a client typed quit().
  • If importing greenlet failed with ImportError, keep the original error message, because sometimes the error originates in setuptools.
  • Changed select.select() to return all the file descriptors signalled, not just the first one.
  • Made thread (and thus monkey patched threads) to spawn Greenlet instances, rather than raw greenlets.

Examples:

  • Updated echoserver.py to use StreamServer.
  • Added geventsendfile.py.
  • Added wsgiserver_ssl.py.

Thanks to Ralf Schmitt for pywsgi, a number of fixes for wsgi, help with baseserver and server modules, improving setup.py and various other patches and suggestions.

Thanks to Uriel Katz for pywsgi patches.

Release 0.12.2 (Mar 2, 2010)

  • Fixed http server to put the listening socket into a non-blocking mode. Contributed by Ralf Schmitt.

Release 0.12.1 (Feb 26, 2010)

  • Removed a symlink from the distribution (that causes pip to fail). Thanks to Brad Clements for reporting it.
  • setup.py: automatically create symlink from build/lib.../gevent/core.so to gevent/core.so.
  • gevent.socket: Improved compatibility with stdlib’s socket:
    • Fixed socket to raise timeout("timed out") rather than simply timeout.
    • Imported _GLOBAL_DEFAULT_TIMEOUT from standard socket module instead of creating a new object.

Release 0.12.0 (Feb 5, 2010)

Release highlights:

  • Added gevent.ssl module.
  • Fixed Windows compatibility (experimental).
  • Improved performance of socket.recv(), socket.send() and similar methods.
  • Added a new module - dns - with synchronous wrappers around libevent’s DNS API.
  • Added core.readwrite_event and socket.wait_readwrite() functions.
  • Fixed several incompatibilities of wsgi module with the WSGI spec.
  • Deprecated pywsgi module.

gevent.wsgi module:

  • Made env["REMOTE_PORT"] into a string.
  • Fixed the server to close the iterator returned by the application.
  • Made wsgi.input object iterable.

gevent.core module:

  • Made DNS functions no longer accept/return IP addresses in dots-and-numbers format. They work with packed IPs now.
  • Made DNS functions no longer accept additional arguments to pass to the callback.
  • Fixed DNS functions to check the return value of the libevent functions and raise IOError if they failed.
  • Added core.dns_err_to_string().
  • Made core.event.cancel not to raise if event_del reports an error. instead, the return code is passed to the caller.
  • Fixed minor issue in string representation of the events.

gevent.socket module:

  • Fixed bug in socket.accept. It could return unwrapped socket instance if socket’s timeout is 0.
  • Fixed socket.sendall implementation never to call underlying socket’s sendall.
  • Fixed gethostbyname() and getaddrinfo() to call the stdlib if the passed hostname has no dots.
  • Fixed getaddrinfo() to filter the results using socktype and proto arguments.
  • Removed getnameinfo() as it didn’t quite match the stdlib interface. Use dns.resolve_reverse() for reverse resolutions.
  • Fixed socket.connect_ex() to use cooperative gethostbyname().
  • Fixed socket.dup() not to call underlying socket’s dup() (which is not available on Windows) but to use Python’s reference counting similar to how the stdlib’s socket implements dup()
  • Added _sock argument to socket‘s constructor. Passing the socket instance as first argument is no longer supported.
  • Fixed socket.connect() to ignore WSAEINVAL on Windows.
  • Fixed socket.connect() to use wait_readwrite() instead of wait_write().
  • Fixed socket.connect() to consult SO_ERROR.
  • Fixed socket.send() and socket.sendall() to support flags argument.
  • Renamed socket_bind_and_listen() to socket.bind_and_listen(). The old name is still available as a deprecated alias.
  • The underlying socket object is now stored as _sock property.
  • Imported the constants and some utility functions from stdlib’s socket into gevent.socket. (Thanks to Matt Goodall for the original patch).
  • Renamed wrap_ssl() to ssl(). (the old name is still available but deprecated)
  • Deprecated connect_tcp() and tcp_server().
  • Added sslerror to socket.__all__.
  • Removed GreenSocket alias for socket class.
  • Moved PyOpenSSL-based implementation of socket.ssl() into gevent.oldssl module. It’s imported into gevent.socket if importing gevent.ssl fails.

Miscellaneous:

  • Fixed Greenlet.spawn_link* and GreenletSet.spawn_link* classmethods not to assume anything about their arguments. (Thanks to Marcus Cavanaugh for pointing that out).
  • Fixed select to clean up properly if event creation fails.
  • Fixed select to raise select.error instead of IOError.
  • Fixed setup.py to proceed with compilation even if libevent version cannot be determined. 1.x.x is assumed in this case.
  • Fixed compatibility of .pyx files with Cython 0.12.0.
  • Renamed arguments for select.select() to what they are called in the stdlib.
  • Removed internal function getLinkedCompleted() from gevent.greenlet.
  • Remove #warning directives from libevent.h. They are not supported by vc90.
  • Removed some deprecated stuff from coros.
  • Internal class Waiter now stores the value if no one’s waiting for it.
  • Added testrunner.py script that replaces a bunch of small scripts that were used before.
  • Removed is_secure attribute from sockets and ssl objects.
  • Made Greenlet not to print a traceback when a not-yet-started greenlet is killed.
  • Added BackdoorServer class to backdoor. Removed backdoor() function and deprecated backdoor_server() function.
  • Removed __getattr__ from socket class.
  • Fixed monkey.patch_socket() not to fail if socket.ssl() is not present in gevent.socket.
  • Added monkey.patch_ssl().
  • Added aggressive argument to monkey.patch_all().
  • Tests from stdlib no longer included in greentest package. Instead, there are number of stubs that import those tests from test package directly and run them in monkey patched environment.
  • Added examples/process.py by Marcus Cavanaugh.

Release 0.11.2 (Dec 10, 2009)

  • Fixed wsgi to unquote environ['PATH_INFO'] before passing to application.
  • Added SERVER_SOFTWARE variable to wsgi environ.
  • Fixed bug in JoinableQueue.task_done() that caused ValueError to be raised incorrectly here.
  • Fixed gevent.socket not to fail with ImportError if Python was not built with ssl support.

Release 0.11.1 (Nov 15, 2009)

  • Fixed bug in select.select() function. Passing non-empty list of write descriptors used to cause this function to fail.
  • Changed setup.py to go ahead with the compilation even if the actual version of libevent cannot be determined (version 1.x.x is assumed in that case).

Contributed by Ludvig Ericson:

  • Fixed wsgi‘s start_response to recognize exc_info argument.
  • Fixed setup.py to look for libevent.dylib rather than .so on Darwin platforms.

Release 0.11.0 (Oct 9, 2009)

  • Fixed timeout bug in joinall(), Greenlet.join(), pool.Pool.join(): if timeout has expired it used to raise Timeout; now it returns silently.
  • Fixed signal() to run the signal handler in a new greenlet; it was run in the Hub greenlet before.
  • Fixed Timeout.start_new(): if passed a Timeout instance, it now calls its start method before returning it.
  • Fixed gevent.monkey to patch threading.local properly.
  • Fixed Queue.empty() and Queue.full() to be compatible with the standard Queue. It tried to take into account the greenlets currently blocking on get/put which was not useful and hard to reason about. Now it simply compares qsize to maxsize, which what the standard Queue does too.
  • Fixed Event to behave exactly like the standard threading.Event:
    • Event.set() does not accept a parameter anymore; it’s now either set or not.
    • Event.get method is gone.
    • Event.set(); Event.clear() used to be a no-op; now it properly wakes up all the waiters.
    • AsyncResult behaves exactly like before, but it does not inherit from Event anymore and does miss clear() method.
  • Renamed internal helpers socket.wait_reader()/socket.wait_writer() to socket.wait_read()/socket.wait_write().
  • Renamed gevent.socket.GreenSocket to gevent.socket.socket. GreenSocket is still available as an alias but will be removed in the future.
  • gevent.core now includes wrappers for evbuffer, evdns, evhttp.
  • Renamed the old gevent.wsgi to gevent.pywsgi.
  • Added a new HTTP server gevent.http module based on libevent-http wrappers.
  • Added a new WSGI server gevent.wsgi module based on gevent.http.
  • Added evdns wrappers to gevent.core and DNS functions to gevent.socket module. Contributed by Jason Toffaletti..
  • Added a few a few options to setup.py to select a libevent library to compile against. Check them out with setup.py -h.
  • Added __all__ to many modules that missed it.
  • Converted the docstrings and the changelog to sphinx/rst markup.
  • Added sphinx/rst documentation. It is available online at http://www.gevent.org.

Release 0.10.0 (Aug 26, 2009)

  • Changed Timeout API in a backward-incompatible way: Timeout.__init__() does not start the timer immediately anymore; Timeout.start() must be called explicitly. A shortcut - Timeout.start_new() - is provided that creates and starts a Timeout.

  • Added gevent.Greenlet class which is a subclass of greenlet that adds a few useful methods join/get/kill/link.

  • spawn() now returns Greenlet instance. The old spawn, which returns py.magic.greenlet instance, can be still accessed as spawn_raw().

    Note

    The implementation of Greenlet is an improvement on proc module, with these bugs fixed:

    • Proc was not a subclass of greenlet which makes getcurrent() useless and using Procs as keys in dict impossible.
    • Proc executes links sequentially, so one could block the rest from being executed. Greenlet executes each link in a new greenlet by default, unless it is set up with Greenlet.rawlink method.
    • Proc cannot be easily subclassed. To subclass Greenlet, override its _run and __init__ methods.
  • Added pool.Pool class with the methods compatible to the standard multiprocessing.pool: apply, map and others. It also has spawn method which is always async and returns a Greenlet instance.

  • Added gevent.event module with 2 classes: Event and AsyncResult. Event is a drop-in replacement for threading.Event, supporting set/wait/get methods. AsyncResult is an extension of Event that supports exception passing via set_exception method.

  • Added queue.JoinableQueue class with task_done and join methods.

  • Renamed core.read and core.write classes to core.read_event and core.write_event.

  • gevent.pywsgi: pulled Mike Barton’s eventlet patches that fix double content-length issue.

  • Fixed setup.py to search more places for system libevent installation. This fixes 64bit CentOS 5.3 installation issues, hopefully covers other platforms as well.

The following items were added to the gevent top level package:

The following items were marked as deprecated:

  • gevent.proc module (wrap_errors helper was moved to util module)
  • gevent.coros.event
  • gevent.coros.Queue and gevent.coros.Channel

Internally, gevent.greenlet was split into a number of modules:

  • gevent.hub provides Hub class and basic utilities, like sleep(); Hub is now a subclass of greenlet.
  • gevent.timeout provides Timeout and with_timeout();
  • gevent.greenlet provides Greenlet class and helpers like joinall() and killall().
  • gevent.rawgreenlet contains the old “polling” versions of joinall and killall (they do not need link functionality and work with any greenlet by polling their status and sleeping in a loop)

Thanks to Jason Toffaletti for reporting the installation issue and providing a test case for WSGI double content-length header bug.

Release 0.9.3 (Aug 3, 2009)

  • Fixed all known bugs in the gevent.queue module and made it 2.4-compatible. LifoQueue and PriorityQueue are implemented as well. gevent.queue will deprecate both coros.Queue and coros.Channel.
  • Fixed Timeout to raise itself by default. TimeoutError is gone. Silent timeout is now created by passing False instead of None.
  • Fixed bug in gevent.select.select() where it could silent the wrong timeout.
  • spawn() and spawn_later() now avoid creating a closure and this decreases spawning time by 50%.
  • kill‘s and killall‘s wait argument was renamed to block. The polling is now implemented by greenlet.join and greenlet.joinall functions and it become more responsive, with gradual increase of sleep time.
  • Renamed proc.RunningProcSet to proc.ProcSet.
  • Added shutdown() function, which blocks until libevent has finished dispatching the events.
  • The return value of event_add and event_del in core.pyx are now checked properly and IOError is raised if they have failed.
  • Fixed backdoor.py, accidentally broken in the previous release.

Release 0.9.2 (Jul 20, 2009)

  • Simplified gevent.socket‘s implementation and fixed SSL bug reported on eventletdev by Cesar Alaniz as well as failures in test_socket_ssl.py.
  • Removed GreenSocket.makeGreenFile; Use socket.socket.makefile() that returns _fileobject and is available on both GreenSocket and GreenSSL. The gevent.socket is still a work in progress.
  • Added new core.active_event class that takes advantage of libevent’s event_active function. core.active_event(func) schedules func to be run in this event loop iteration as opposed to core.timer(0, ...) which schedules an event to be run in the next iteration. active_event is now used throughout the library wherever core.timer(0, ....) was previously used. This results in spawn() being at least 20% faster compared to release 0.9.1 and twice as fast compared to eventlet. (The results are obtained with bench_spawn.py script in greentest/ directory)
  • Added boolean parameter wait to kill() and killall() functions. If set to True, it makes the function block until the greenlet(s) is actually dead. By default, kill() and killall() are asynchronous, i.e. they don’t unschedule the current greenlet.
  • Added a few new properties to gevent.core.event: fd, events, events_str and flags. It also has __enter__ and __exit__ now, so it can be used as a context manager. event‘s callback signature has changed from (event, fd, evtype) to (event, evtype).
  • Fixed Hub‘s mainloop to never return successfully as this will screw up main greenlet’s switch() call. Instead of returning it raises DispatchExit.
  • Added reinit() function - wrapper for libevent’s event_reinit. This function is a must have at least for daemons, as it fixes epoll and some others eventloops to work after fork.
  • Trying to use gevent in another thread will now raise an exception immediately, since it’s not implemented.
  • Added a few more convenience methods spawn_link[exception/value] to proc.RunningProcSet.
  • Fixed setup.py not to depend on setuptools.
  • Removed gevent.timeout. Use gevent.Timeout.

Release 0.9.1 (Jul 9, 2009)

  • Fixed compilation with libevent-1.3. Thanks to Litao Wei for reporting the problem.
  • Fixed Hub to recover silently after event_dispatch() failures (I’ve seen this happen after fork even though event_reinit() is called as necessary). The end result is that fork() now works more reliably, as detected by test_socketserver.py - it used to fail occasionally, now it does not.
  • Reorganized the package, most of the stuff from gevent/__init__.py was moved to gevent/greenlet.py. gevent/__init__.py imports some of it back but not everything.
  • Renamed gevent.timeout to gevent.Timeout. The old name is available as an alias.
  • Fixed a few bugs in queue.Queue. Added test_queue.py from standard tests to check how good is queue.Queue a replacement for a standard Queue (not good at all, timeouts in queue.Queue.put() don’t work yet)
  • monkey now patches ssl module when on 2.6 (very limited support).
  • Improved compatibility with Python 2.6 and Python 2.4.
  • Greenlet installed from PyPI (without py.magic prefix) is properly recognized now.
  • core.pyx was accidentally left out of the source package, it’s included now.
  • GreenSocket now wraps a socket object from _socket module rather than from socket.

Release 0.9.0 (Jul 8, 2009)

Started as eventlet 0.8.11 fork, with the intention to support only libevent as a backend. Compared to eventlet, this version has a much simpler API and implementation and a few severe bugs fixed, namely

  • Full duplex in sockets, i.e. read() and write() on the same fd do not cancel one another.
  • The GreenSocket.close method does not hang as it could with eventlet.

There’s a test in my repo of eventlet that reproduces both of them: http://bitbucket.org/denis/eventlet/src/tip/greentest/test__socket.py

Besides having less bugs and less code to care about the goals of the fork are:

  • Piggy-back on libevent as much as possible (use its http and dns code).
  • Use the interfaces and conventions from the standard Python library where possible.