Desenvolvendo protótipos para startups com Python e web2py

No dia 18 eu palestrei na semana global do empreendedorismo, la na Plug'n work. A idéia da palestra foi mostrar para os empreendedores que estão começando a desenvolver suas idéias uma maneira de desenvolver seus protótipos (ou até mesmo um MVP) utilizando Python, web2py, bootstrap e o browser.

Além de apresentar Python e ressaltar sua facilidade, assim como todo o poder do web2py para este nicho de público, eu tive a intenção de focar em uma opinião pessoal que é a minha repulsa pelo termo "Sócio técnico" e como isso soa como enganação e é claro mostrar como qualquer empreendedor que saiba usar um computador e pelo menos tenha noção de estrutura de dados (ja tenha usado uma planilha excel) é capaz de desenvolver seu próprio protótipo utilizando o web2py.

Pretendo melhorar este material e quem sabe transformar em um vídeo e tambem estou disponível para dar a mesma palestra em outros eventos, universidades etc..

Seguem os slides.

Este video é parte da aula 4 do cursodepython.com.br

web2py and Redis Queue

RQ (Redis Queue)

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web(2py) stack easily.

http://python-rq.org

web2py

Free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applications. Written and programmable in Python.

http://www.web2py.com

Queueing jobs with RQ and web2py

web2py as many other web frameworks works in a request -> response environment, which means that there is a lifetime for things to be done. This lifetime we call "request time", it is the time between the client requests a resource (i.e hits an url of our app or post a form) and the time that the server gives the response back to the client (i.e: The server sends html, json or any other kind of response).

The problem with this is the fact that we have a time-out and the user does not want to wait for tasks to be done, I mean in example for creating image thumbnails, users have to upload a picture and then wait for the thumbnail to be created to have a response from server. Or in the case of sending an email, user fill a contact form and have to wait for the message to be sent. It can take a long time and sometimes it will fail.

The solution is to enqueue that jobs on background and then watch its results to give a response to the user, this response can be given through a websocket or ajax long pooling. (I will not cover this here)

Setting up

  • install Redis
    • In debian based linuxes you can do: sudo apt-get install redis-server
  • Install RQ (redis queue)
    • sudo pip install rq

case 1 : Sending email in background

User will fill our contact form and then click in submit, instead of sending the email we are going to enqueue the email to be sent by the redis queue.

1. In your models create your Queue object (also you need to have the mail settings)


    from gluon.tools import Mail
    mail = Mail()
    mail.settings.server = "smtp.google.com:587"
    mail.settings.sender = "you@gmail.com"
    mail.settings.login = "you:yourpassword"


```models/queue.py
from redis import Redis
from rq import Queue
q = Queue(connection=Redis())

The above will use the default Redis connection port to localhost, take a look at RQ docs if you need to set another redis server.

2. In your controller create the contact action which returns a form.


    def contact():
        form = SQLFORM.factory(Field("name"), Field("message"))
        if form.accepts(request):
            # enqueue the email to be sent!
            q.enqueue(mail.send,
                      to="you@gmail.com",
                      subject="%(name)s contacted you" % form.vars,
                      message=form.vars.message)
            # do whatever you want
            response.flash = "email successfully sent!"

## case 2 : Creating an image thumbnail

User will upload a picture and you are going to create a THUMBNAIL and store the thumbnail in /static/thumbs folder

#### 1. Define some models

```models/db.py
Picture = db.define_table("pictures",
    Field("name"),
    Field("picture", "upload")
)

    from redis import Redis
    from rq import Queue
    q = Queue(connection=Redis())

#### 2. Create the form

```controllers/default.py
# requires PIL to be installed
# sudo apt-get install python-imaging
from gluon.contrib.imageutils import THUMB

def add_picture():
    form = SQLFORM(Picture, submit_button="send")
    if form.process().accepted:
        #enqueue thumbnail to be created
        q.enqueue(THUMB, form.vars.picture)

Put the workers to work

On the cases above we just enqueued tasks to be executed by the workers, now we need the worker running.

web2py environment

The worker should run under the web2py environment, because we are using web2py modules to send emails and create the thumbnail, so the RQ worker should be started with this script.

1. Create the web2py RQ worker

/some/path/web2py-rq.py

import sys
from rq import Queue, Connection, Worker

# Provide queue names to listen to as arguments to this script,
# similar to rqworker
with Connection():
    qs = map(Queue, sys.argv[1:]) or [Queue()]
    w = Worker(qs)
    w.work()

Start the above worker under web2py environment

cd /path/to/web2py
python web2py.py -S yourappname -M -R /some/path/web2py-rq.py

With the above worker running the enqueued tasks will be executed and then worker will keep listening for new tasks.

You can also put the worker to run in backgroungm for this you shoud use nohup python web2py.py -S yourappname -M -R /some/path/web2py-rq.py & or even better you can put this to run under the supervidord

with the worker running you should see this console:

python web2py/web2py.py -S app -M -R /projects/web2py-rq.py 
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2012
Version 2.4.1-alpha.2+timestamp.2012.12.28.16.18.51
Database drivers available: SQLite(sqlite3), MySQL(pymysql), PostgreSQL(pg8000), IMAP(imaplib)
[2012-12-31 00:33] DEBUG: worker: Registering birth of worker precise64.15755
[2012-12-31 00:33] INFO: worker: RQ worker started, version 0.3.2
[2012-12-31 00:33] INFO: worker: 
[2012-12-31 00:33] INFO: worker: *** Listening on default...
[2012-12-31 00:34] INFO: worker: default: send(to='someone@gmail.com', message='blah', subject='testing') (a069b2c6-f908-4806-8534-b00c43996cf4)

Monitoring

RQ has some nice ways for monitoring the jobs by command-line or by its dashboard.

command line:

To see what queues exist and what workers are active, just type rqinfo:

$ rqinfo
high       |██████████████████████████ 20
low        |██████████████ 12
default    |█████████ 8
3 queues, 45 jobs total

Bricktop.19233 idle: low
Bricktop.19232 idle: high, default, low
Bricktop.18349 idle: default
3 workers, 3 queues

As you can see it is possible to start many workers.

Dashboard

The easiest way is probably to use the RQ dashboard, a separately distributed tool, which is a lightweight webbased monitor frontend for RQ, which looks like this:

RQ dashboard https://github.com/nvie/rq-dashboard

dash

microblog app

microblog app

Este tutorial foi criado para o evento RuPy Brasil em parceria com a ZNC Sistemas.

O download do app pode ser feito em: Download pacote w2p

O tutorial em PDF: Tutorial em PDF

Tutorial: Criando um microblog app

Agora pretendo aproveitar que o blog tem mais espaço que o PDF para detalhar um pouco mais o app de microblog e também implementar algumas funcionalidades extra.

CONTINUE:

Search form with web2py

Quick and dirty search form example

Considering models/db.py

status_options = {"0": "pending", "1": "confirmed", "3": "canceled"}

db.define_table("orders",
    Field("id_buyer", "reference auth_user"),
    Field("order_date", "date"),
    Field("status",
          requires=IS_IN_SET(status_options),
          represent= lambda value, row: status_options[value]
          ),
    Field("obs","text")
)

And the search function controllers/default.py

import datetime

@auth.requires_login()
def index():
    # default values to keep the form when submitted
    # if you do not want defaults set all below to None

    status_default = request.vars.status
    date_initial_default = \
        datetime.datetime.strptime(request.vars.date_initial, "%Y-%m-%d") \
            if request.vars.date_inicial else None
    date_final_default = \
        datetime.datetime.strptime(request.vars.date_final, "%Y-%m-%d") \
            if request.vars.date_final else None
    obs_default = request.vars.obs


    # The search form created with .factory
    form = SQLFORM.factory(
                  Field("status",
                        default=status_default
                        requires=IS_EMPTY_OR(
                            IS_IN_SET(status_options, zero="-- All --")
                        ),
                  ),
                  Field("date_initial", "date", default=date_initial_default),
                  Field("date_final", "date", default=date_final_default),
                  Field("obs", default=obs_default),
                  formstyle='divs',
                  submit_button="Search",
                  )

    # The base query to fetch all orders of the current logged user
    query = db.orders.id_buyer == auth.user_id                  

    # testing if the form was accepted              
    if form.process().accepted:
        # gathering form submitted values
        status = form.vars.status
        date_initial = form.vars.date_initial
        date_final = form.vars.date_final
        obs = form.vars.obs

        # more dynamic conditions in to query
        if status:
            query &= db.orders.status == status
        if date_initial:
            query &= db.orders.order_date >= date_initial
        if date_final:
            query &= db.orders.order_date <= date_final
        if obs:
            # A simple text search with %like%
            query &= db.orders.obs.like("%%%s%%" % obs)            

    count = db(query).count()
    results = db(query).select(orderby=~db.orders.data)
    msg = T("%s registers" % count )
    return dict(form=form, msg=msg, results=results) 

Optionally you can create a view file in views/default/index.html

{{extend 'layout.html'}}
{{=form}}
<hr />
{{=msg}}
{{=results}}

the end result

Download the app: http://dl.dropbox.com/u/830444/web2py.app.busca.w2p

If you need a better and complex search engine I recommend Whoosh.

App news reading (portuguese)

models/db.py

db = DAL("sqlite://news.sqlite")
db.define_table("noticias",
    Field("titulo"),
    Field("texto", "text"),
    Field("data", "datetime")
    )

controllers/default.py

def escrever():
    form = SQLFORM(db.noticias)
    if form.process().accepted:
        redirect(URL("listar"))
    return dict(form=form)

def listar():
    noticias = db(db.noticias).select(orderby=~db.noticias.data)
    return dict(noticias=noticias)

def ler_noticia():
    id_noticia = request.args(0) or redirect(URL("listar"))
    # caso não passe um id retorna para /listar
    noticia = db.noticias[id_noticia]
    return dict(noticia=noticia)

views/default/listar.html

{{extend "layout.html"}}

<ul>
{{for noticia in noticias:}}
<li> 
    <a href="{{=URL("default", "ler_noticia", args=noticia.id)}}"> 
    {{=noticia.titulo}} 
    </a>
 </li>
{{pass}}
</ul>

views/default/ler_noticia.html

{{extend "layout.html"}}

<h1>
    <a href="{{=URL("default", "ler_noticia", args=noticia.id)}}">
        {{=noticia.titulo}} 
    </a>
</h1>
<p>
{{=XML(noticia.texto)}}
</p>

views/default/escrever.html

{{extend "layout.html"}}

<h1> escreva uma noticia</h1>
{{=form}}

WEb2py 2.0

I am now a member of Python Software Foundation

Now that's official, So I can blog about it here.

Massimo Di Pierro, The lead developer/creator of web2py framework which I started to use/contribute 3 years ago. Nominated me and Mariano Reingart to become Python Software Foundation members. In Aug 13th, 2012 PSF had an election, and we were accepted as PSF elected members.

I am honored to be nominated by Massimo, and also very happy to have been accepted. Not just a personal recognition for me, but also a recognition for all the work done with web2py, and also highlighted the importance of a web2py within the Python community.

I guess that the PSF members who voted have read my CV and obviously my historical analysis did not take into account only the work I do with web2py, but also the work I've been doing to promote and strengthen the use of the Python language in my country with my participation in events as well as advocating and teaching courses about the language and promoting it in companies where I give consulting services.

However, taking into account the fact that my dedication in the Python community is mostly focused on the promotion and development with/of web2py. I assume that this election has taken this into consideration.


Here it comes the reproduction from PSFBLOG

Just the other day the Python Software Foundation held an election, the second and final one of the year, and the results are in! 18 new members were introduced, and the membership approved three new sponsor members. Please join us in welcoming all of them!
Candidates for PSF membership are nominated by an existing member for their work in the Python community. The membership is comprised of people from around the world and from many areas of the community.
These new members are selected from many different areas of the Python community. While some members are known for their contributions of code, many are known for their work to grow their local and regional communities. Some members are known for their work in educational workshops and conferences. It takes a diverse membership to ensure the success of a foundation steering a diverse community, so we're happy to have members of all types from all areas, both geographically and within the Python world.

Please join us in welcoming all of the new members to the Foundation!

Nick Barcet
Dana Bauer
James Blair
Thierry Carrez
Anand Chittipothu
Antonio Cuni
Anne Gentle
Noufal Ibrahim
Vish Ishaya
Christopher MacGown
Dave Malcolm
Joshua McKenty
Mark McLoughlin
Mariano Reingart
Bruno Rocha
Monty Taylor
Dean Troyer
Vicky Twomey-Lee

The following sponsor members were approved:

DreamHost
Globo.com
Hood Media GmbH

For the full PSF membership roster, please see http://www.python.org/psf/members/


Thank you Massimo, and PSF members. I really hope (and I will do my best) to retribute this nomination with continuous work for Python Community.

web2py - manage users and membership in the same form

As requested by user of Stack Overflow.
http://stackoverflow.com/questions/11992749/web2py-how-edit-user-profile-and-membership-in-one-view

How to manage users and memberships at the same form

NOTE: You have to register the first admin user first, because to manage users and memberships we require to be admin

For the purpose of the example we are going to use the file controllers/default.py accessible at the url localhost:8000/YOURAPP/default

A grid to list your users

this is the list admins will see when hit http://..../default/list_users

The user list grid

1 - Put on the default.py file

#@auth.requires_membership("admin") # uncomment to enable security 
def list_users():
    btn = lambda row: A("Edit", _href=URL('manage_user', args=row.auth_user.id))
    db.auth_user.edit = Field.Virtual(btn)
    rows = db(db.auth_user).select()
    headers = ["ID", "Name", "Last Name", "Email", "Edit"]
    fields = ['id', 'first_name', 'last_name', "email", "edit"]
    table = TABLE(THEAD(TR(*[B(header) for header in headers])),
                  TBODY(*[TR(*[TD(row[field]) for field in fields]) \
                        for row in rows]))
    table["_class"] = "table table-striped table-bordered table-condensed"
    return dict(table=table)

With generic views will see this

2 - The edit links to manage_users

Now accessing http://..../default/list_users you are going to see the grid showing all users, now if you click on the edit link on grid it goes to manage_user function we defined on btn = lambda row: A("Edit", _href=URL('manage_user', args=row.auth_user.id))

Create this two functions in the same controller

The user form
#@auth.requires_membership("admin") # uncomment to enable security 
def manage_user():
    user_id = request.args(0) or redirect(URL('list_users'))
    form = SQLFORM(db.auth_user, user_id).process()
    membership_panel = LOAD(request.controller,
                            'manage_membership.html',
                             args=[user_id],
                             ajax=True)
    return dict(form=form,membership_panel=membership_panel)

On the above function we are going to create two objects form which is the form to edit the user object, also we create membership_panel which is an ajax panel to load the manage_membership inside it and ajax managed.

note: that this function takes user_id from request.args(0) then if it is not provided it redirects back to the list_users

The membership panel
#@auth.requires_membership("admin") # uncomment to enable security 
def manage_membership():
    user_id = request.args(0) or redirect(URL('list_users'))
    db.auth_membership.user_id.default = int(user_id)
    db.auth_membership.user_id.writable = False
    form = SQLFORM.grid(db.auth_membership.user_id == user_id,
                       args=[user_id],
                       searchable=False,
                       deletable=False,
                       details=False,
                       selectable=False,
                       csv=False,
                       user_signature=False)
    return form

note that on the manage_membership we are returning the form directly, so we can input it inside the ajax panel membership_panel

The manage_user view

3 - Create an html file in YOURAPP/views/default/manage_user.html

{{extend 'layout.html'}}
<h4> Edit The user </h4>
{{=form}}
<hr>
<h4> User membership </h4>
{{=membership_panel}}

The end result

User Form

Add membership


Done, using web2py 2.0 (trunk)

Lazy DAL - Attempt 3 - Pbreit

based on Pbreit request

On Wed, Aug 15, 2012 at 2:32 PM, pbreit wrote:
What would it take to set it up such that models are defined in mostly the same way as now but in "module" files and then imports are done in controllers/functions that need access to the table.

This file goes on modules/mymodels.py

# -*- coding: utf-8 -*-

from gluon.dal import DAL, Field
from gluon import current

DBURI = "sqlite://....."

TABLE_DEFINITIONS = {
    "owners": {
        "fields": [Field("name")],
        "kwargs": dict(format="%(name)s")       
    },
    "cars": {
        "fields": [Field("name"),
                   Field("owner", "reference owner")],
        "kwargs": dict(format="%(name)s")       
    }
}

class Models(object):
    def __init__(self):
        self.db = DAL(DBURI)

    @property 
    def tables(self):
        return self.db.tables

    def __call__(self, *args, **kwargs):
        return self.db(*args, **kwargs)

    def table_definer(self, tablename):   
      if not tablename in self.db.tables:
          fields = TABLE_DEFINITIONS.get(tablename, {}).get('fields', [])
          kwargs = TABLE_DEFINITIONS.get(tablename, {}).get("kwargs", {})
          return self.db.define_table(tablename, *fields, **kwargs)
      return self.db[tablename]

    def __getattr__(self, key):
      if hasattr(self.db, key):
          return getattr(self.db, key)
      elif key not in TABLE_DEFINITIONS.keys():
          raise AttributeError("attr not found")
      else:
          return self.table_definer(key)

Now in any controller controllers/default.py

from mymodels import Models()
db = Models()

def list_owners():
    rows = db(db.owners).select()
    return dict(rows=rows)

I tested and works, the only caveat is that you are not going to use db.define_table but you will put your model definitions on TABLE_DEFINITIONS dict

websockets com tornado, web2py, Python, jQuery

Comet messaging with Tornado and web2py

(in Portuguese)