Microservices with Python, RabbitMQ and Nameko

"Micro-services is the new black" - Splitting the project in to independently scalable services is the currently the best option to ensure the evolution of the code. In Python there is a Framework called "Nameko" which makes it very easy and powerful.

Micro services

The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. - M. Fowler

I recommend reading the Fowler's posts to understand the theory behind it.

Ok I so what does it mean?

In brief a Micro Service Architecture exists when your system is divided in small (single context bound) responsibilities blocks, those blocks doesn't know each other, they only have a common point of communication, generally a message queue, and does know the communication protocol and interfaces.

Give me a real-life example

The code is available on github: http://github.com/rochacbruno/nameko-example take a look at service and api folders for more info.

Consider you have an REST API, that API has an endpoint receiving some data and you need to perform some kind of computation with that data, instead of blocking the caller you can do it asynchronously, return an status "OK - Your request will be processed" to the caller and do it in a background task.

Also you want to send an email notification when the computation is finished without blocking the main computing process, so it is better to delegate the "email sending" to another service.

Scenario

enter image description here

Show me the code!

Lets create the system to understand it in practice.

Environment

We need an environment with:

  • A running RabbitMQ
  • Python VirtualEnv for services
  • Python VirtualEnv for API

Rabbit

The easiest way to have a RabbitMQ in development environment is running its official docker container, considering you have Docker installed run:

docker run -d --hostname my-rabbit --name some-rabbit -p 15672:15672 -p 5672:5672 rabbitmq:3-management

Go to the browser and access http://localhost:15672 using credentials guest:guest if you can login to RabbitMQ dashboard it means you have it running locally for development.

enter image description here

The Service environment

Now lets create the Micro Services to consume our tasks. We'll have a service for computing and another for mail, follow the steps.

In a shell create the root project directory

$ mkdir myproject
$ cd myproject

Create and activate a virtualenv (you can also use virtualenv-wrapper)

$ virtualenv service_env
$ source service_env/bin/activate

Install nameko framework and yagmail

(service_env)$ pip install nameko
(service_env)$ pip install yagmail

The service code

Now having that virtualenv prepared (consider you can run service in a server and API in another) lets code the nameko RPC Services.

We are going to put both services in a single python module, but you can also split in separate modules and also run them in separate servers if needed.

In a file called service.py

import yagmail
from nameko.rpc import rpc, RpcProxy


class Mail(object):
    name = "mail"

    @rpc
    def send(self, to, subject, contents):
        yag = yagmail.SMTP('myname@gmail.com', 'mypassword')
        # read the above credentials from a safe place.
        # Tip: take a look at Dynaconf setting module
        yag.send(to=to.encode('utf-8'), 
                 subject=subject.encode('utf-8'), 
                 contents=[contents.encode('utf-8')])


class Compute(object):
    name = "compute"
    mail = RpcProxy('mail')    

    @rpc
    def compute(self, operation, value, other, email):
        operations = {'sum': lambda x, y: int(x) + int(y),
                      'mul': lambda x, y: int(x) * int(y),
                      'div': lambda x, y: int(x) / int(y),
                      'sub': lambda x, y: int(x) - int(y)}
        try:
            result = operations[operation](value, other)
        except Exception as e:
            self.mail.send.async(email, "An error occurred", str(e))
            raise
        else:
            self.mail.send.async(
                email, 
                "Your operation is complete!", 
                "The result is: %s" % result
            )
            return result

Now with the above services definition we need to run it as a Nameko RPC service.

NOTE: We are going to run it in a console and leave it running, but in production it is recommended to put the service to run using supervisord or an alternative.

Run the service and let it running in a shell

(service_env)$ nameko run service --broker amqp://guest:guest@localhost
starting services: mail, compute
Connected to amqp://guest:**@127.0.0.1:5672//
Connected to amqp://guest:**@127.0.0.1:5672//

Testing it

Go to another shell (with the same virtenv) and test it using nameko shell

(service_env)$ nameko shell --broker amqp://guest:guest@localhost
Nameko Python 2.7.9 (default, Apr  2 2015, 15:33:21) 
[GCC 4.9.2] shell on linux2
Broker: amqp://guest:guest@localhost
>>>

You are now in the RPC client testing shell exposing the n.rpc object, play with it

>>> n.rpc.mail.send("name@email.com", "testing", "Just testing")

The above should sent an email and we can also call compute service to test it, note that it also spawns an async mail sending with result.

>>> n.rpc.compute.compute('sum', 30, 10, "name@email.com")
40
>>> n.rpc.compute.compute('sub', 30, 10, "name@email.com")
20
>>> n.rpc.compute.compute('mul', 30, 10, "name@email.com")
300
>>> n.rpc.compute.compute('div', 30, 10, "name@email.com")
3

Calling the micro-service through the API

In a different shell (or even a different server) prepare the API environment

Create and activate a virtualenv (you can also use virtualenv-wrapper)

$ virtualenv api_env
$ source api_env/bin/activate

Install Nameko, Flask and Flasgger

(api_env)$ pip install nameko
(api_env)$ pip install flask
(api_env)$ pip install flasgger

NOTE: In api you dont need the yagmail because it is service responsability

Lets say you have the following code in a file api.py

from flask import Flask, request
from flasgger import Swagger
from nameko.standalone.rpc import ClusterRpcProxy

app = Flask(__name__)
Swagger(app)
CONFIG = {'AMQP_URI': "amqp://guest:guest@localhost"}


@app.route('/compute', methods=['POST'])
def compute():
    """
    Micro Service Based Compute and Mail API
    This API is made with Flask, Flasgger and Nameko
    ---
    parameters:
      - name: body
        in: body
        required: true
        schema:
          id: data
          properties:
            operation:
              type: string
              enum:
                - sum
                - mul
                - sub
                - div
            email:
              type: string
            value:
              type: integer
            other:
              type: integer
    responses:
      200:
        description: Please wait the calculation, you'll receive an email with results
    """
    operation = request.json.get('operation')
    value = request.json.get('value')
    other = request.json.get('other')
    email = request.json.get('email')
    msg = "Please wait the calculation, you'll receive an email with results"
    subject = "API Notification"
    with ClusterRpcProxy(CONFIG) as rpc:
        # asynchronously spawning and email notification
        rpc.mail.send.async(email, subject, msg)
        # asynchronously spawning the compute task
        result = rpc.compute.compute.async(operation, value, other, email)
        return msg, 200

app.run(debug=True)

Put the above API to run in a different shell or server

(api_env) $ python api.py 
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

and then access the url http://localhost:5000/apidocs/index.html you will see the Flasgger UI and you can interact with the api and start producing tasks on queue to the service to consume.

[image]

NOTE: You can see the shell where service is running for logging, prints and error messages. You can also access the RabbitMQ dashboard to see if there is some message in process there.

There is a lot of more advanced things you can do with Nameko framework you can find more information on https://nameko.readthedocs.org/en/stable/

Let's Micro Serve!

ESEngine - Elasticsearch Object Doctype Mapper for Python

What is ESEngine

esengine - The Elasticsearch Object Doctype Mapper

PyPI versions downloads Travis CI Coverage Status Code Health



ESEngine is an ODM (Object Doctype Mapper) heavily inspired by MongoEngine, developed with the idea that you have to "Know well your Elastic queries and then write them as Python objects"

You extend the esengine.Document class defining a bunch of fields and meta-attributes and you can use that model to instantiate documents and perform queries on ElasticSearch.

ESEngine is MIT licensed and is open source available at http://github.com/catholabs/esengine

The documentation is currently only a README full of examples in http://catholabs.github.io/esengine/ and also the DocString that can be read using Epydoc in http://catholabs.github.io/esengine/docs/

How it works?

Firstly you need an Elasticsearch Python Client, we recommend using the official one pip install elasticsearch, and then you can define your models using ESEngine objects.

# myproject/models.py
from elasticsearch import Elasticsearch
from esengine import Document, StringField, BooleanField

class Person(Document):
    # meta attributes
    _index = 'myproject'
    _doctype = 'person'

    # default client instance
    _es = Elasticsearch()  # optional, can be passed lazily or can be a callable 

    # field definitions
    name = StringField()
    active = BooleanField()

Person.init()

NOTE: The init() calling will initialize the index/doctype mappings and settings, this part can be omitted and then Elastic Search will try to create this by introspection when the first document is indexed.

With the model definition in a file like myproject/models.py we can now use the model class Person to Index(insert), edit, delete and of course search documents.

In a Python console:

>>> from myproject.models import Person

Indexing:

>>> user = Person(name=”Bruno”, active=True)
>>> user.save()
# or simply
>>> user =  Person.create(name=”Bruno”, active=True)

Updating

>>> user.active = False
>>> user.save()
# or simply
>>> user.update(active=False)

Filtering multiple documents

>>> users = Person.filter(active=True)
[ ResultSet generator… a list of active users ]

Bulk update

>>> users.update(active=False)

Performing raw queries (recommended)

>>> query = {“query”: {“match_all”: {}}, “sort”: “name”} 
>>> Person.search(query=query, size=10)

Querying using Payload helpers (better to create dynamic queries)

>>> from esengine import Payload, Query
>>> query = Query.match_all() 
>>> Payload(model=Person, query=query, sort=”name”).search(size=10)

Deleting documents

>>> user = Person.get(id=123)
>>> user.delete()
# or simply
>>> Person.delete_by_id(123)
# or in bulk
>>> users = Person.filter(active=False)
>>> Person.delete_all(users)
# ou simply
>>> Person.delete_by_query({“query”: …. })

You can find more examples in https://github.com/catholabs/esengine

Currently ESEngine is being used in 3 of Catholabs production projects and is reaching a nice level of performance and abstraction.

If you use ElasticSearch with Python or want to learn more about it you can follow the readme on github feel free to open issues or collaborating by Pull Requests Pull Requests :)

Let's Search!

Flasgger - API playground with Flask and Swagger UI

What is Swagger?

Swagger is a simple yet powerful representation of your RESTful API. With the largest ecosystem of API tooling on the planet, thousands of developers are supporting Swagger in almost every modern programming language and deployment environment. With a Swagger-enabled API, you get interactive documentation, client SDK generation and discoverability.

What is Swagger UI?

Swagger UI is a dependency-free collection of HTML, Javascript, and CSS assets that dynamically generate beautiful documentation and sandbox from a Swagger-compliant API. Because Swagger UI has no dependencies, you can host it in any server environment, or on your local machine. Head over to the online demo to see what it looks like for any publically accessible Swagger definition.

What is Flask? (duhhhh!!)

Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions. Why it is awesome? because it is simple yet powerful, talking is cheaper look at the code!

from flask import Flask, jsonify, request
app = Flask(__name__)

@app.route('my_awesome_api', methods=['POST'])
def my_awesome_endpoint():
    data = request.json
    return jsonify(data=data, meta={"status": "ok"})

app.run()

Run the above script and then you can start posting to the API

curl -XPOST http://localhost:5000/my_awesome_api -d '{"python": "is awesome"}'
{
    "data": {"python": "is awesome"},
    "meta": {"status": "ok"}
}

What is Flasgger?

Flasgger is a Flask extension to help the creation of Flask APIs with documentation and live playground powered by SwaggerUI. You can define your API structure using YAML files and Flasgger creates all the specifications for you and you can use the same schema to validate the data.

GITHUB REPO: https://github.com/rochacbruno/flasgger

Install it

pip install flasgger

Create your app

You can put API specs directly in docstrings

import random
from flask import Flask, jsonify, request
from flasgger import Swagger

app = Flask(__name__)
Swagger(app)

@app.route('/api/<string:language>/', methods=['GET'])
def index(language):
    """
    This is the language awesomeness API
    Call this api passing a language name and get back its features
    ---
    tags:
      - Awesomeness Language API
    parameters:
      - name: language
        in: path
        type: string
        required: true
        description: The language name
      - name: size
        in: query
        type: integer
        description: size of awesomeness
    responses:
      500:
        description: Error The language is not awesome!
      200:
        description: A language with its awesomeness
        schema:
          id: awesome
          properties:
            language:
              type: string
              description: The language name
              default: Lua
            features:
              type: array
              description: The awesomeness list
              items:
                type: string
              default: ["perfect", "simple", "lovely"]

    """

    language = language.lower().strip()
    features = [
        "awesome", "great", "dynamic", 
        "simple", "powerful", "amazing", 
        "perfect", "beauty", "lovely"
    ]
    size = int(request.args.get('size', 1))
    if language in ['php', 'vb', 'visualbasic', 'actionscript']:
        return "An error occurred, invalid language for awesomeness", 500
    return jsonify(
        language=language,
        features=random.sample(features, size)
    )


app.run(debug=True)

Try it!

Now run your app and access http://localhost:5000/apidocs/index.html and you will play with Swagger UI!

Screenshot Flasgger

NOTE: All the default urls can be changed in configuration.

It is also possible to use a separate file for specs

Create your api specification in a separated YML file

In a file index.yml put the specs definitions

    This is the language awesomeness API
    Call this api passing a language name and get back its features
    ---
    tags:
      - Awesomeness Language API
    parameters:
      - name: language
        in: path
        type: string
        required: true
        description: The language name
      - name: size
        in: query
        type: integer
        description: size of awesomeness
    responses:
      500:
        description: Error The language is not awesome!
      200:
        description: A language with its awesomeness
        schema:
          id: awesome
          properties:
            language:
              type: string
              description: The language name
              default: Lua
            features:
              type: array
              description: The awesomeness list
              items:
                type: string
              default: ["perfect", "simple", "lovely"]

and then change the code to read from it using swag_from decorator

import random
from flask import Flask, jsonify, request
from flasgger import Swagger
from flasgger.utils import swag_from

app = Flask(__name__)
Swagger(app)

@app.route('/api/<string:language>/', methods=['GET'])
@swag_from('index.yml')
def index(language):
    language = language.lower().strip()
    features = [
        "awesome", "great", "dynamic", 
        "simple", "powerful", "amazing", 
        "perfect", "beauty", "lovely"
    ]
    size = int(request.args.get('size', 1))
    if language in ['php', 'vb', 'visualbasic', 'actionscript']:
        return "An error occurred, invalid language for awesomeness", 500
    return jsonify(
        language=language,
        features=random.sample(features, size)
    )


app.run(debug=True)

validation

If you put the specs in a separate file it is also possible to use the same specs to validate the input

from flasgger.utils import swag_from, validate, ValidationError

@app.route('/api/<string:language>/', methods=['GET'])
@swag_from('index.yml')
def index(language):
    ...
    try:
        validate(data, 'awesome', 'index.yml', root=__file__)
    except ValidationError as e:
        return "Validation Error: %s" % e, 400
    ...

More information

You can find more information and some examples in the github repository

Contribute

Please share your thoughts about ir, open issues, give ideas and PullRequests are always welcome!

Let's Swag!

Dynaconf - Let your settings to be Dynamic

Dynaconf

dynaconf - The dynamic configurator for your Python Project

MIT License PyPI downloads Travis CI Coverage Status Code Health

dynaconf is an OSM (Object Settings Mapper) it can read settings variables from a set of different data stores such as python settings files, environment variables, redis, memcached, ini files, json files, yaml files and you can customize dynaconf loaders to read from wherever you want. (maybe you really want to read from xml files ughh?)

GITHUB REPO: https://github.com/rochacbruno/dynaconf



What is Dynaconf?

Dynaconf is a common point of access to settings variables, you import only one object in your project and from that object you can access settings variables from Python settings file, from environment variables, from parsed yaml, ini, json or xml files, from datastores as Redis and MongoDB or from wherever your need if you write a simple dynaconf loader.

How it works

Install it

pip install dynaconf

Use it

from dynaconf import settings
print settings.SOME_VARIABLE
or
print settings.get('SOME_VARIABLE')

By default Dynaconf will try to use a file called settings.py on the root of your project, if you place that file there all upper case variables will be read

You can also replace the file exporting an environment variable pointing to the module or location for the settings file.

# using module name
export DYNACONF_SETTINGS=myproject.production_settings
# or using location path
export DYNACONF_SETTINGS=/etc/myprogram/settings.py

Doing that when you use from dynaconf import settings the variables will be read from that file.

So how it is Dynamic?

Now think you have your program done and you want to deploy to a certain infrastructure for testing or maybe different deployment, you don't need to rewrite the settings file. Just export some variables to your environment.

export DYNACONF_MYSQL_HOST=myserver.com

Now in your project you can do:

from dynaconf import settings
print settings.MYSQL_HOST
myserver.com

The default prefix for exported envvars is by default DYNACONF_ but you also can change it if needed.

But what if I have some typed values to export?

You can also define type casting when exporting and those types will be used to parse the values.

export DYNACONF_NUMBER='@int 123'
export DYNACONF_FLOAT='@float 12.2'
export DYNACONF_FLAG='@bool yes'
export DYNACONF_FLAG2='@bool disabled'
export DYNACONF_LIST='@json [1, 2, 3, 4]'
export DYNACONF_DICT='@json {"name": "Bruno"}'

Now you can read all those values from your project and it will be loaded with correct type casting.

from dynaconf import settings

type(settings.NUMBER)
int

type(settings.FLOAT)
float

type(settings.FLAG)
bool

print settings.FLAG2 == False
True

print settings.LIST[1]
2

print settings.DICT['name']
Bruno

Nice! But I don't want to use envvars because I use autoscaling and I want my machines to share a settings environment how to do it?

Redis

Go to your settings file (default settings.py) and put

# connection
REDIS_FOR_DYNACONF = {
    'host': 'localhost',
    'port': 6379,
    'db': 0
}

# and loader
LOADERS_FOR_DYNACONF = [
    'dynaconf.loaders.env_loader',
    'dynaconf.loaders.redis_loader'
]

Now you can store settings variables directly in Redis using a hash named by default DYNACONF_DYNACONF

If you don't want want to write directly you can use the Redis writer helper in a python REPL. (ipython as example)

from dynaconf.utils import redis_writer
from dynaconf import settings

redis_writer.write(settings, name='Bruno', mysql_host='localhost', MYSQL_PORT=1234)

And the above will be store in Redis as a hash int the form.

DYNACONF_DYNACONF:
    NAME='Bruno'
    MYSQL_HOST='localhost'
    PORT='@int 1234'

And of course you can now read those variables in the project, all the casting wildcards also works on Redis but if you want to skip type casting, write as string intead of PORT=1234 use PORT='1234' as redis stores everything as string anyway.

There is more

Dynaconf has support for using different namespaces in the same project, you can also write your own loaders, you can find more information on the repository https://github.com/rochacbruno/dynaconf

Contribute

All contributions are very welcome!!

Acknowledgements

Dynaconf was inspired by Flask app config and also by Django settings module.

Dealing with linked containers dependency in docker-compose

In docker-compose a common problem is starting services and daemons in containers that depends on services running on linked containers, in example: your app depends on elasticsearch but it is not ready when the app container is started. Solution is to use a wait script.

I knew that docker-compose team is working on adding a WAIT parameter, but while it is not ready we can use a wait script to load our services.

The process is simple, your container will execute a shell script that will try to execute a HEALTH CHECK many times you want before starting the main command. The health check, the command, the sleep and how many loops to try will be defined in environment variables

This is a program that needs to access an elasticsearch server located in 'elastic' host (that will be mapped by compose

access_elastic_search.py

from datetime import datetime 
from elasticsearch import Elasticsearch
es = Elasticsearch('elastic')
es.index(index="my-index", doc_type="test-type", id=42, body={"any": "data", "timestamp": datetime.now()})
print es.get(index="my-index", doc_type="test-type", id=42)

docker-compose.yml

elastic:
    image: elasticsearch
    command: elasticsearch -Des.node.name="YourNodeName"
    ports:
       - "9200:9200"

app:
    image: ubuntu:14.04
    command: wait_to_start
    working_dir: /src
    links:
        - elastic:elastic
    volumes:
        - .:/src
    environment:
        - WAIT_COMMAND=[ $(curl --write-out %{http_code} --silent --output /dev/null http://elastic:9200/_cat/health?h=st) = 200 ]
        - WAIT_START_CMD=python access_elastic_search.py
        - WAIT_SLEEP=2
        - WAIT_LOOPS=10

wait_to_start script

 #!/bin/bash

echo $WAIT_COMMAND
echo $WAIT_START_CMD

is_ready() {
    eval "$WAIT_COMMAND"
}

# wait until is ready
i=0
while ! is_ready; do
    i=`expr $i + 1`
    if [ $i -ge $WAIT_LOOPS ]; then
        echo "$(date) - still not ready, giving up"
        exit 1
    fi
    echo "$(date) - waiting to be ready"
    sleep $WAIT_SLEEP
done

#start the script
exec $WAIT_START_CMD

Now when you start your environment with docker-compose up the app container will start and execute wait_to_start script, which will perform the test defined in WAIT_COMMAND environment variable and will retry until the test succeeds, so it will end executing the program defined in WAIT_START_CMD

Full code on gist: https://gist.github.com/rochacbruno/bdcad83367593fd52005

This code was made with StackOverflow help

Oportunidade para Programador Python / Data Scientist na Catho

Você gosta de trabalhar com inteligência aplicada?
Não tem medo de aprender novas tecnologias?
Venha trabalhar conosco!
 
A equipe de Inovação da Catho procura por alguém com paixão por conhecimento e espírito inovador.
Nosso foco é produzir novas tecnologias que irão ajudar pessoas a encontrarem as melhores vagas
e empresas a encontrarem os melhores profissionais.
 
Temos dois perfis de vaga:
 

Perfil cientifico, para efetuar analise e desenvolvimento de modelos

e ferramentas voltadas para mineracao de dados e inteligencia artifical;

 
Áreas de conhecimento importantes no nosso dia-a-dia:
  • Machine Learning
  • Sistemas de Recomendação
  • Modelos estatísticos
  • Recuperação de Informação
 

Perfil desenvolvedor, para desenvolver aplicaçoes baseadas em data mining e big data;

 
Desenvolvimento de software:
  • Ambiente: Linux, Git, Github
  • Programação: Python
  • Banco de dados: MongoDB, MySQL, Postgres
  • Web: Flask, Tornado, Javascript, HTML/CSS
Computaçao aplicada:
  • Máquinas de busca: Elasticsearch, Solr
  • Computação em nuvem & alto desempenho: AWS, NewRelic, Otimizaçao de performance
  • Cloud & Big Data
  • Infra: Fabric, NewRelic, MMS
  • Cloud: Amazon
  • Big data: Amazon SWF
 
Requisitos:
 
  • Ser fluente em uma linguagem de programação
  • Ter autonomia para mergulhar em problemas
  • Trabalhar bem em equipe
  • Ter espírito de pesquisador
 
Mais Informações:
 
  • Localização: Tamboré, Barueri/SP
  • Contrato: CLT Full
  • Jornada: Integral/Flexível
  • Benefícios: Plano de Saúde e Dental, VR/VA, Vale transporte,
  • Seguro de vida, Estacionamento, Convênio SESC
  • Remuneração: A combinar
 
Contato:
 
Envie seu resumo profissional para catholabs@catho.com

What The Flask - Série de 6 artigos + tutorial para aprender desenvolvimento web com Python e Flask

What The Flask

6 passos para ser um Flask ninja!

Olá, nesta semana comecei a publicar a série What The Flask lá no PythonClub.

http://res.cloudinary.com/diu8g9l0s/image/upload/v1400201393/pythonclub/logo_275x130.png

para quem ainda não conhece, o PythonClub é um blog colaborativo feito pela comunidade e hospedado no github. Colabore você também!

Nesta série de 6 artigos/tutoriais pretendo abordar de maneira bem detalhada o desenvolvimento web com o framework Flask.

Depois de mais de um ano desenvolvendo projetos profissionais com o Flask e adquirindo experiência também no desenvolvimento do projeto open source Quokka CMS resolvi compartilhar algumas dicas para facilitar a vida de quem pretende começar a desenvolver para web com Python.

A série What The Flask será dividida nos seguintes capítulos.

  1. Hello Flask: Introdução ao desenvolvimento web com Flask
  2. Flask patterns: boas práticas na estrutura de aplicações Flask
  3. Plug & Use: extensões essenciais para iniciar seu projeto
  4. DRY: Criando aplicativos reusáveis com Blueprints
  5. from flask.ext import magic: Criando extensões para o Flask e para o Jinja2
  6. Run Flask Run: "deploiando" seu app nos principais web servers e na nuvem.

Hello Flask

Parte 1 - Introdução ao desenvolvimento web com Flask

Conhecendo o Flask

Quick and Dirty Tutorial: Desenvolvendo um aplicativo de notícias

Para acompanhar a série completa acesse o PythonClub

Comentários, dúvidas, sugestões e formulário a respeito do curso online e do livro de Flask também estão no lá no final do artigo no PythonClub. :)

Usando o Flask Cache

Como framework o Flask não tem nenhuma funcionalidade de cache embutida, porém existe a cache API do werkzeug e uma excelente extensão para prover essas funcionalidades de cache para suas aplicacoes Flask, esta extensão foi criado pelo @thadeusb e é bastante fácil de implementar e utilizar.

instalando

Em sua env instale pelo PyPI (recommended)

pip install Flask-Cache  

Você também pode instalar diretamente através do github se quiser pegar a última versao de desenvolvimento.

pip install https://github.com/thadeusb/flask-cache/tarball/master

Configurando

Existe uma série de chaves de configuração que podem ser setadas, mas a mais importante é a que define o backend de cache CACHE_TYPE.

A chave CACHE_TYPE pode ser uma string resolvendo um caminho de importação de uma classe que implemente a api de cache do werkzeug, mas existem alguns alias que mapeam as implemetações default que estão no werkzeug.contrib.cache

Por padrão o CACHE_TYPE é NUll o que significa que o cache não terá efeito, então você precisa escolher um backend para o cache.

chave | Cache type

  • null | Sem Cache - NullCache
  • simple | Usará pickle em memória e é recomendado apenas em servidor single process durante o desenvolvimento
  • memcached | Requer pylibmc ou memcached e precisa das configs do memcached no settings
  • redis | Requer redis, werkzeug 0.7 e configurações do Redis no settings
  • filesystem | O mesmo que o simple, porém armazena o pickle em filesystem ao inves da memoria
  • gaememcached | Para o Google AppEngine
  • saslmemcached | Mesmo que o memcached mas para instalações em SASL - requer pylibmc

Todas as opções e outras chaves de config em http://pythonhosted.org/Flask-Cache/#configuring-flask-cache

App Flask simples

em um arquivo nomeado app.py

import time
from flask import Flask

app = Flask(__name__)

@app.route("/")
def view():
    return time.ctime()

if __name__ == "__main__":
    app.run(port=5000, debug=True, host='0.0.0.0')

Execute com python app.py e abra a url http://localhost:5000 no seu browser, aperte F5 (refresh) para ver a data e hora correntes.

Habilitando o cache para a view

agora vamos habilitar o cache para esta pequena app e evitar que a hora seja atualizada a cada request. (neste exemplo usamos a data atual mas imagine que isso poderia ser o carregamento de um grande conjunto de dados ou cálculos complexos)

em um arquivo nomeado cached_app.py

import time
from flask import Flask

# importe a extensao
from flask.ext.cache import Cache   

app = Flask(__name__)

# defina a configuração do cache (isso pode ser feito em um arquivo de settings)
app.config['CACHE_TYPE'] = 'simple'

# instancie o cache e atribua a sua aplicação
app.cache = Cache(app)   

@app.route("/")
@app.cache.cached(timeout=300)  # cache this view for 5 minutes
def cached_view():
    return time.ctime()

if __name__ == "__main__":
    app.run(port=5000, debug=True, host='0.0.0.0')

Execute com python cached_app.py e abra http://localhost:5000 em seu browser e aperte F5 (refresh) para ver a data e hora cacheadas durante 5 minutos.

O decorator @cache.cached utiliza o request.path para a view como chave de cache, se por algum motivo você precisar especificar uma chave diferente poderá passar o argumento key_prefix para o decorator. Neste caso, se o seu argumento incluir um placeholder "%s" ele será substituido pelo request.path.

O exemplo acima é a mais simples e regular app Flask com o uso de cache, mas se sua app é projetada usando app factories, blueprints, class based views ou views separadas em arquivos diferentes você precisará usar abordagens avançadas.

Cacheando funções

O mesmo decorator cached pode ser usado para cachear funções (que não são views), neste caso será preciso especificar um key_prefix, caso contrário o request.path será usado e então poderá cair em conflito quando existir mais de uma função cacheada.

Neste exemplo usaremos o módulo this para extrair citações do Zen do Python.

Em um arquivo nomeado cached_function_app.py

import time
import random

from this import s, d
from string import translate, maketrans

from flask.ext.cache import Cache
from flask import Flask

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

@app.cache.cached(timeout=10, key_prefix="current_time")
def get_current_time():
    return time.ctime()

def random_zen_quote():
    """Pega uma citação aleatória do Zen do Python""" 
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@app.route("/")
def zen():
    return """
    <ul>
        <li><strong>Com cache:</strong> {cached}</li>
        <li><strong>Sem cache:</strong> {not_cached}</li>
    </ul>
    """.format(
        cached=get_current_time(),
        not_cached=random_zen_quote()
    )

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Execute com python cached_function_app.py acesse http://localhost:5000 apertando F5 (refresh) você verá que a data atual é cacheada mas a citação continua sendo atualizada, você pode alternar o cache para ver como fica.

def get_current_time():
    return time.ctime()

@app.cache.cached(timeout=10, key_prefix="zen_quote")
def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@app.route("/")
def zen():
    return """
    <ul>
        <li><strong>Sem cache:</strong> {cached}</li>
        <li><strong>Com cache:</strong> {not_cached}</li>
    </ul>
    """.format(
        cached=get_current_time(),
        not_cached=random_zen_quote()
    )

NOTE: por ter usado o módulo this você verá algumas citações serem impressas em seu terminal mas isso não tem nenhum problema.

Cacheando views modulares

agora um exemplo com views separadas em um arquivo diferente para melhor organização

Em um pasta chamada app coloque 3 arquivos __init__.py, app.py e views.py

app/__init__.py um arquivo vazio

app/views.py

import time
import random
from this import s, d
from string import translate, maketrans

def get_current_time():
    return time.ctime()

def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

def zen_view():
    return """
    <h1>No cache por 10 segundos!</h1>
    <ul>
        <li>{time}</li>
        <li>{quote}</li>
    </ul>
    """.format(
        time=get_current_time(),
        quote=random_zen_quote()
    )

Como pode ver no exemplo acims definimos a view em um arquivo separado. Para evitar import circular não seria recomendado usar @app.route nem o @app.cache portanto esta view será agnostica com relação a app pois iremos registrar seu mapeamento de urls e seu cache no arquivo principal da app.

Este tipo de estrutura é necessária quando se tem muitas views e é preciso organizar em vários arquivos separados.

NOTE: Para uma melhor organização é recomendado o uso de blueprints que será explicado em seguida.

app/app.py

agora na app principal importaremos as views e explicitamente decoraremos com o cache e também registraremos a regra de url.

from flask import Flask
from flask.ext.cache import Cache
from views import zen_view

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

# aplique o cache usando o velho estilo de uso dos decorators
cached_zen_view = app.cache.cached(timeout=10)(zen_view)

# registre o mapeamento de url
app.add_url_rule("/", view_func=cached_zen_view)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

NOTE: Tambem é possivel colocar o cache em um arquivo separado como veremos em seguida.

Cacheando views em Blueprints

Como foi mencionado anteriormente, a melhor maneira de organizar uma app Flask é com o uso de Blueprints, que é uma maneira de criar ' meta-apps' que serão conectadas ao app principal no momento da inicialização, O problema aqui é que o Blueprint é feito pensando em criar módulos reutilizaveis por diferentes apps, portanto o controle do cache precisa ser delegado dinamicamente.

Para evitar import circular criaremos uma instancia de cache em um arquivo separado (considere usar application factory se estiver criando algo mais complexo)

Crie uma pasta chamada blueprint_app com os seguintes arquivos

cached_blueprint_app/
├── app.py
├── cache.py
├── blueprints
│   ├── __init__.py
│   └── zen_blueprint.py
└── __init__.py

No cache.py

from flask.ext.cache import Cache    
cache = Cache()

Criamos uma instancia burra do Cache, ela ainda não foi inicializada mas já pode ser usada, no futuro ela será inicializada junto com a app e estará disponivel quando a view for chamada. Para isso reimportaremos a mesma instancia de cache e chamaremos o metodo init_app.

Um blueprint básico blueprints/zen_blueprint.py

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint
from cache import cache

zen = Blueprint('zen', __name__)

def get_current_time():
    return time.ctime()

def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@zen.route("/")
@cache.cached(timeout=20)
def zen_view():
    return """
    <h1>Cacheado por 20 segundos!</h1>
    <ul>
        <li>{time}</li>
        <li>{quote}</li>
    </ul>
    """.format(
        time=get_current_time(),
        quote=random_zen_quote()
    )

NOTE: Em uma aplicação real é recomendado separar mais o blueprint em arquivos diferents para views, urls e helpers, para isso você poderá promover este blueprint para um pacote Python.

A app principal app.py

from flask import Flask

from blueprints.zen_blueprint import zen
from cache import cache

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
cache.init_app(app)

app.register_blueprint(zen)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Repare que criamos a instancia burra do cache no arquivo cache.py e usamos esta instancia para decorar as views do blueprint, então o cache foi inicializado no app.py utilizando o método init_app. Isso e possivel por conta do ciclo de inicialização do Flask e também pelo fato da extensão Flask-Cache ter sido criada pensando nesses casos. Se um dia pretender escrever sua própria extensão para o Flask aconselho dar uma olhada no código fonte da Flask-Cache.

Execute com python cached_blueprint_app/app.py acesse http://localhost:5000 e a view será cacheada por 20 segundos

Cacheando MethodViews

Vamos usar o mesmo exemplo cached_blueprint_app transformado a view zen_view em uma MethodView

Altere o zen_blueprint.py para:

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint
from flask.views import MethodView
from cache import cache

zen = Blueprint('zen', __name__)

class ZenView(MethodView):

    @cache.cached(30)
    def get(self):
        return """
        <h1>Cacheado por 30 segundos!</h1>
        <ul>
            <li>{time}</li>
            <li>{quote}</li>
        </ul>
        """.format(
            time=self.get_current_time(),
            quote=self.random_zen_quote()
        )

    @staticmethod
    def get_current_time():
        return time.ctime()

    @staticmethod
    def random_zen_quote():
        transtable = maketrans("".join(d.keys()), "".join(d.values()))
        return random.choice(translate(s, transtable).split("\n")[2:])


zen.add_url_rule("/", view_func=ZenView.as_view('zen'))

MethodViews mapeiam os métodos HTTP como GET, POST e DELETE para métodos da classe como get, post, delete etc, então neste caso criamos um método chamado get e decoramos com o @cache.cached.

NOTE: Você geralmente não pode usar decorators em métodos individuais de MethodViews, porém o Flask-Cache foi implementado de forma que permite este uso.

alternativamente você pode querer cachear todos os métodos de uma mesma view, para isso basta cachear o dispatch_request ou melhor ainda cachear a view completa usando um decorator explicito.

Cacheando o dispatcher
class ZenView(MethodView):
    @cache.cached(timeout=30)
    def dispatch_request(self):
        return super(ZenView, self).dispatch_request()

    ...
Cacheando a view com decorator explicito (recomendado)
zen = Blueprint('zen', __name__)

class ZenView(MethodView):
    ...

cached_zen_view = cache.cached(timeout=50)(ZenView.as_view('zen'))
zen.add_url_rule("/", view_func=cached_zen_view)

Cacheando blocos de template

O Flask-Cache vem com uma template tag que permite cachear blocos de template, vamos mudar a ZenView e faze-la renderizar um template com Jinja2

No zen_blueprint.py

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint, render_template
from flask.views import MethodView

zen = Blueprint('zen', __name__)

class ZenView(MethodView):

    def get(self):
        return render_template(
            'zen.html',
            get_random_quote=self.random_zen_quote
        )

    @staticmethod
    def get_current_time():
        return time.ctime()

    @staticmethod
    def random_zen_quote():
        transtable = maketrans("".join(d.keys()), "".join(d.values()))
        return random.choice(translate(s, transtable).split("\n")[2:])

zen.add_url_rule("/", view_func=ZenView.as_view('zen'))

Agora teremos que criar o template cached_blueprint_app/templates/zen.html

<h3> Zen of Python Aleatório </h3>
<strong>{{get_random_quote()}}</strong>

Execute com python cached_blueprint_app/app.py e abra http://localhost:5000 você verá a citação ser atualizada a cada request, vamos evitar isso cacheando durante 30 segundos.

Change the zen.html template

{% cache 30 %}
<h3> Zen of Python Aleatório </h3>
<strong>{{get_random_quote()}}</strong>
{% endcache %}

Salve o template e de o refresh varias vezes e você verá que a citação estará cacheada por 30 segundos.

Cacheando funções com argumentos variaveis usando o decorator memoize

as vezes as views ou funções recebem argumentos através do mapeamento de urls ou diretamente passados, você pode querer cachear utilizando esses argumentos como chave do cache para ter caches diferentes para cada chamada. No Flask-Cache tem o decorator memoize para isso.

NOTE: Para funções que não recebem argumentos cached e memoize terão o mesmo efeito

Com uma app simples memoize_app.py

import time
from flask.ext.cache import Cache
from flask import Flask

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

@app.cache.memoize(timeout=5)
def get_current_time_and_name(name):
    return "%s - %s" % (name, time.ctime())

@app.route("/<name>")
def view(name):
    return get_current_time_and_name(name)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Rode com python memoize_app.py abra http://localhost:5000/seunome e repare que a chamada é cacheada separadamente para cada argumento <name> passado.

Cacheando objetos diretamente

As vezes não podemos usar decorators e precisamos cachear objetos diretamente.

Dentro de uma view ou blueprint é possivel usar o current_app

from flask import current_app

def some_function():
    cached = current_app.cache.get('a_key')
    if cached:
        return cached
    result = do_some_stuff()
    current_app.cache.set('a_key', result, timeout=300)
    return result

Ou se estiver usando uma instancia separada de cache (recomendado)

from cache import cache

def function():
    cached = cache.get('a_key')
    if cached:
        return cached
    result = do_some_stuff()
    cache.set('a_key', result, timeout=300)
    return result

Limpando o cache

Você pode criar um script para limpar o cache ou uma função para ser chamada quando preciso.

from flask.ext.cache import Cache    
from yourapp import app
cache = Cache()

def main():
    cache.init_app(app)

    with app.app_context():
        cache.clear()

if __name__ == '__main__':
    main()

AVISO: Em alguns backends a limpeza completa do cache não é suportada. Se não estiver usando key_prefix em alguns backends como redis irá fazer com que todo o database seja apagado.

Existe uma série de exemplos e uma API bem documentada no site do Flask-Cache http://pythonhosted.org/Flask-Cache/ também é possivel criar um novo backend de cache seguindo este exemplo.

Using Flask Cache

As a micro framework Flask does not have built-in cache functionality, however, there is werkzeug cache API and an excellent extension to provide its caching functionality to your Flask apps, that extension was created by @thadeusb and is very easy to implement and use.

installing

In your env install it via PyPI (recommended)

pip install Flask-Cache  

You can also install it directly from source code if you need recent changes or bugfixes

pip install https://github.com/thadeusb/flask-cache/tarball/master

Configuring

There is a set of configuration keys you can put in your app settings, but the most important is the cache backend defined by the CACHE_TYPE key.

The cache type resolves to an import string which needs to be an object implementing the werkzeug cache api, but there is some aliases to the werkzeug.contrib.cache implementations

By default the CACHE_TYPE is Null which means that your app will have no cache, so you need to choose of the options below:

  • null | No Cache - NullCache
  • simple | Will use in memory pickle and is recommended only for single process development server
  • memcached | Requires pylibmc or memcached and requires memcached configuration in settings
  • redis | Requires redis, werkzeug 0.7 and redis configuration in settings
  • filesystem | The same as simple but stores the pickle in a cache_dir
  • gaememcached | For Google AppEngine
  • saslmemcached | The same as memcached but for SASL - pylibmc required

Full options and config variables are in http://pythonhosted.org/Flask-Cache/#configuring-flask-cache

A Simple Flask app

a file named app.py

import time
from flask import Flask

app = Flask(__name__)

@app.route("/")
def view():
    return time.ctime()

if __name__ == "__main__":
    app.run(port=5000, debug=True, host='0.0.0.0')

Run the above with python app.py and open http://localhost:5000 in your browser and hit F5 (refresh) to see the current date and time.

Enabling Flask-Cache for views

Now we want to enable caching on that small application to avoid the refresh of the current time (for this example we are using current time as return but imagine that it could be a large dataset or huge calculations)

a file named cached_app.py

import time
from flask import Flask

# import the flask extension
from flask.ext.cache import Cache   

app = Flask(__name__)

# define the cache config keys, remember that it can be done in a settings file
app.config['CACHE_TYPE'] = 'simple'

# register the cache instance and binds it on to your app 
app.cache = Cache(app)   

@app.route("/")
@app.cache.cached(timeout=300)  # cache this view for 5 minutes
def cached_view():
    return time.ctime()

if __name__ == "__main__":
    app.run(port=5000, debug=True, host='0.0.0.0')

Run the above with python cached_app.py and open http://localhost:5000 in your browser and hit F5 (refresh) to see that the current date and time is now cached for 5 minutes.

@cache.cached decorator takes the request.path for that view and use this as cache key, if for any reason you need a different key you can pass a key_prefix argument to the decorator. In this case if you pass a key_prefix containing the %s placeholder it will be replaced by the current request.path

The above is the simplest and regular example of Flask app and the use of cache, but, if your app is designed using application factories, blueprints, class based views or views located in different modules you will need to use advanced approach.

Caching regular functions

The same cached decorator can be used to cache regular functions, but in this case you will need to specify the key_prefix argument, otherwise it will use the request.path which can lead to conflicts if you have many cached functions.

For this example we are going to use the this module and extract a random quote from the Zen of Python.

A file named cached_function_app.py

import time
import random

from this import s, d
from string import translate, maketrans

from flask.ext.cache import Cache
from flask import Flask

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

@app.cache.cached(timeout=10, key_prefix="current_time")
def get_current_time():
    return time.ctime()

def random_zen_quote():
    """Pick a random quote from the Zen of Python""" 
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@app.route("/")
def zen():
    return """
    <ul>
        <li><strong>It is cached:</strong> {cached}</li>
        <li><strong>It is not cached:</strong> {not_cached}</li>
    </ul>
    """.format(
        cached=get_current_time(),
        not_cached=random_zen_quote()
    )

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Now running python cached_function_app.py and opening http://localhost:5000 when hitting F5 to refresh you will see the current time cached for 5 minutes and the random quote updated, you can switch the cache just to see the efect.

def get_current_time():
    return time.ctime()

@app.cache.cached(timeout=10, key_prefix="zen_quote")
def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@app.route("/")
def zen():
    return """
    <ul>
        <li><strong>It is not cached:</strong> {cached}</li>
        <li><strong>It is cached:</strong> {not_cached}</li>
    </ul>
    """.format(
        cached=get_current_time(),
        not_cached=random_zen_quote()
    )

NOTE: Because we are importing the this module for the example, you will see the Zen quotes in your flask terminal, but there is no problem with this.

Caching modular views

Now an example when you have your app splitted in two or more files for better organization

in a folder called app put 3 files__init__.py, app.py and views.py

app/__init__.py is an empty file

app/views.py

import time
import random
from this import s, d
from string import translate, maketrans

def get_current_time():
    return time.ctime()

def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

def zen_view():
    return """
    <h1>Cached for 10 seconds!</h1>
    <ul>
        <li>{time}</li>
        <li>{quote}</li>
    </ul>
    """.format(
        time=get_current_time(),
        quote=random_zen_quote()
    )

as you can see the above file defined view functions, as it it a separated file, to avoid circular imports we are not recommended to use @app.route neither @app.cache so this views will be app agnostic and we are going to register its url rules and caching in the main app file.

That kind of structure is needed when your app has too many views and want a better organization.

NOTE: For better organization the mostly recommended pattern is Blueprints which I will explain further.

app/app.py

Now in the main app we need to import our views, explicitly decorate for caching and also register its urls.

from flask import Flask
from flask.ext.cache import Cache
from views import zen_view

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

# explicitly apply the cache in the old-style decoration way
cached_zen_view = app.cache.cached(timeout=10)(zen_view)

# explicitly register the cached view url mapping
app.add_url_rule("/", view_func=cached_zen_view)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

NOTE: You can also separate the cache instance in a different file for lazy initialization as we are going to see in the next example

Caching Blueprint views

As mentioned before, the best pattern to follow in Flask applications is the Blueprint pattern which is a way to create separated 'meta-apps' that will be connected to your main application in the time of initialization, the problem here is that Blueprints are meant to be reusable by many different applications, so the delegation of cache control should be dynamized.

In order to avoid circular imports you will want to create your cache instance separate from your application instance (you may want to consider switching to the app factory module if you are building something more complex).

Create a folder called blueprint_app with the following structure

cached_blueprint_app/
├── app.py
├── cache.py
├── blueprints
│   ├── __init__.py
│   └── zen_blueprint.py
└── __init__.py

The cache.py

from flask.ext.cache import Cache    
cache = Cache()

we can create a dummy lazy cache instance, that will be initialized in the future when the view will be called. For that in the app we are going to reimport the same cache instance and call init_app method.

The basic blueprints/zen_blueprint.py

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint
from cache import cache

zen = Blueprint('zen', __name__)

def get_current_time():
    return time.ctime()

def random_zen_quote():
    transtable = maketrans("".join(d.keys()), "".join(d.values()))
    return random.choice(translate(s, transtable).split("\n")[2:])

@zen.route("/")
@cache.cached(timeout=20)
def zen_view():
    return """
    <h1>Cached for 20 seconds!</h1>
    <ul>
        <li>{time}</li>
        <li>{quote}</li>
    </ul>
    """.format(
        time=get_current_time(),
        quote=random_zen_quote()
    )

NOTE: In a real application you will want to modularize it separating the views, helpers etc and promoting your blueprint to a Python package.

The main app.py

from flask import Flask

from blueprints.zen_blueprint import zen
from cache import cache

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
cache.init_app(app)

app.register_blueprint(zen)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Notice that we created a dummy instance of cache in cache.py and then used that instance to decorate the blueprints views, then the cache was initialized in app.py with init_app method. That is possible because of the Flask initialization cycle and the excellent implementation in Flask-Cache extension that takes care of this case, if you plan to write yor own Flask extension take a look at the Flask-Cache source code.

Run the application by calling python cached_blueprint_app/app.py and open http://localhost:5000 to see the blueprint view cached for 20 seconds.

Caching MethodView

Lets use the same cached_blueprint_app example but turning the zen_view in to a MethodView

Change your zen_blueprint.py to:

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint
from flask.views import MethodView
from cache import cache

zen = Blueprint('zen', __name__)

class ZenView(MethodView):

    @cache.cached(30)
    def get(self):
        return """
        <h1>Cached for 30 seconds!</h1>
        <ul>
            <li>{time}</li>
            <li>{quote}</li>
        </ul>
        """.format(
            time=self.get_current_time(),
            quote=self.random_zen_quote()
        )

    @staticmethod
    def get_current_time():
        return time.ctime()

    @staticmethod
    def random_zen_quote():
        transtable = maketrans("".join(d.keys()), "".join(d.values()))
        return random.choice(translate(s, transtable).split("\n")[2:])


zen.add_url_rule("/", view_func=ZenView.as_view('zen'))

Method views maps HTTP method names as GET, POST, DELETE to the view methos as get, post, delete etc, So all we needed to do is to create a method called get and decorate it with @cache.cached decorator.

NOTE: Due to the implicit self from the caller’s perspective you cannot use regular view decorators on the individual methods of the view however, Flask-Cache is one exception because its implementation allow the use of cached decorator in individual methods. Keep this in mind.

Alternativelly you may want to cache all the methods in a view, for that you can cache the dispatch_request method or even better you can decorate the whole view.

Caching the dispatcher
class ZenView(MethodView):
    @cache.cached(timeout=30)
    def dispatch_request(self):
        return super(ZenView, self).dispatch_request()

    ...
Caching the whole view (recommended)
zen = Blueprint('zen', __name__)

class ZenView(MethodView):
    ...

cached_zen_view = cache.cached(timeout=50)(ZenView.as_view('zen'))
zen.add_url_rule("/", view_func=cached_zen_view)

Caching template blocks

Flask cache comes with a template tag able to cache template blocks, lets change our ZenView to use a Jinja2 template

in zen_blueprint.py

import time
import random
from this import s, d
from string import translate, maketrans
from flask import Blueprint, render_template
from flask.views import MethodView

zen = Blueprint('zen', __name__)

class ZenView(MethodView):

    def get(self):
        return render_template(
            'zen.html',
            get_random_quote=self.random_zen_quote
        )

    @staticmethod
    def get_current_time():
        return time.ctime()

    @staticmethod
    def random_zen_quote():
        transtable = maketrans("".join(d.keys()), "".join(d.values()))
        return random.choice(translate(s, transtable).split("\n")[2:])

zen.add_url_rule("/", view_func=ZenView.as_view('zen'))

Now we need to create a template file in cached_blueprint_app/templates/zen.html

<h3> Random Zen of Python </h3>
<strong>{{get_random_quote()}}</strong>

Running the application with python cached_blueprint_app/app.py and opening http://localhost:5000 you will see a random quote refreshed every time you push F5, lets cache it for 30 second.

Change the zen.html template

{% cache 30 %}
<h3> Random Zen of Python </h3>
<strong>{{get_random_quote()}}</strong>
{% endcache %}

Now save the file and refresh to see the content cached for 30 seconds.

Caching functions and views with variant arguments using memoize decorator

Sometimes yout views and functions receives arguments which can come from url mapping or directly to the function call, yiou may want to cache the view or funtion and use the arguments as keys to cache its different results, Flask-Cache has a different decorator for doing that.

NOTE: With functions that do not receive arguments, cached() and memoize() are effectively the same.

Now with a simple application memoize_app.py

import time
from flask.ext.cache import Cache
from flask import Flask

app = Flask(__name__)
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)

@app.cache.memoize(timeout=5)
def get_current_time_and_name(name):
    return "%s - %s" % (name, time.ctime())

@app.route("/<name>")
def view(name):
    return get_current_time_and_name(name)

if __name__ == "__main__":
    app.run(debug=True, port=5000, host='0.0.0.0')

Now run python memoize_app.py and open http://localhost:5000/yourname and note that the function will be cached for each different name you pass as argument in the url.

Caching arbitrary objects

There are some times when decorators cannot be used and you need to explicitly set or get some thing on the cache.

Inside a view or a blueprint you can use current_app

from flask import current_app

def some_function():
    cached = current_app.cache.get('a_key')
    if cached:
        return cached
    result = do_some_stuff()
    current_app.cache.set('a_key', result, timeout=300)
    return result

Or if using a separete cache instance you can do this directly

from cache import cache

def function():
    cached = cache.get('a_key')
    if cached:
        return cached
    result = do_some_stuff()
    cache.set('a_key', result, timeout=300)
    return result

Clearing the cache

You can create a script to clear the cache, or a function to use it when needed

from flask.ext.cache import Cache    
from yourapp import app
cache = Cache()

def main():
    cache.init_app(app)

    with app.app_context():
        cache.clear()

if __name__ == '__main__':
    main()

WARNING: Some backend implementation do not support completely clearing the case. Also, if you’re not using key prefix, some implementation (e.g. Redis) will flush the whole database. Make sure you’re not storing any other data in your caching database.

There is a lot of examples and well documented API in flask-Cache website http://pythonhosted.org/Flask-Cache/ you can also create your own cache backend following the examples in the Flask-Cache docs.

Python Meme de Ano Novo #2014pythonmeme

Python Meme de ano Novo

(não achei melhor tradução para o título)

Inspirado pelos posts de Alex Clark e Daniel Greenfield.

Qual o framework, aplicação ou biblioteca mais legal que você descobriu neste ano?

  • xmltodict
    Eu escrevi um post a respeito, e esta é definitivamente a melhor maneira de ler e escrever arquivos XML. Eu ainda não sei nada a respeito de performance mas já utilizei em produção não tive nennum problema. é o requests para XML.
  • Kivy + QPython
    Eu já conhecia o Kivy mas ainda não havia utilizado em nenhum projeto, neste ano tive a oportunidade de iniciar um projeto o Kivy já se tornou minha primeira opção para projetos desktop/mobile, poderia dizer que o Kivy é o novo "Tkinter" e desejo que se torne "padrão" para desktop/mobile em Python. O Qpython é uma distribuição Python para Android e já vem com suporte a Kivy e Pygame, além disso roda também o Django e o Flask numa boa!
  • MongoDB Aggregation Framework
    Pipelines são d+!!! se eu puder, nunca mais quero usar SQL para criar relatórios financeiros.
  • MongoEngine
    Eu ja usava muito o PyMongo desde 2011, então comecei a trabalhar muito com o Django ORM (que eu não gosto), mas a sintaxe dele se tornou natural e foi bom encontrar a mesma sintaxe para o Mongo.

Qual nova técnica de programação você aprendeu neste ano?

  • Descriptors
    Eu já sabia o básico dos descriptors pois escrevi vários exemplos para o cursodepython.com.br, porém só agora em 2013 que tive a oportunidade de usar em produção no Quokka CMS. Aprendi muito sobre descriptios e sobre Python
  • SSE
    Não é uma técnica e sim uma tecnologia/padrão. Nunca tinha usado e aprendi como implementar o backend e o frontend e tambem passei por vários problemas em produção.

Qual projeto Open Source que você mais contribuiu neste ano? e o que você fez?

  • Quokka CMS
    Iniciei o projeto em Julho de 2013 e 3 meses depois ja tinha mais de 500 starts no github (graças ao hacker news) e vários contribuidores. Agora já coloquei 3 sites em produção usando Quokka e seus módulos cart, classes e fundraising. Quokka foi bastante inspirado em outro projeto Open Source que trabalhei em 2013.
  • Opps CMS
    Eu comecei o ano de 2013 com um novo emprego na YACOWS e com o novo emprego um projeto muito bacana chamado OPPS (OPen Publishing System). Nós desenvolvemos o Opps em 3 meses e criamos 3 grandes portais de alto trafégo com muitos plugins legais para liveblog, feeds, analytics etc. Eu criei vários desses apps para o Opps (Enquetes, Promoções, Goalserve api de esportes, Feeds)
  • Python Pagseguro Pag Seguro é o gateway de pagamentos mais utilizado no Brasil e eu criei uma lib para integrar Python com a API v2 do pagseguro
  • Flask GoogleMaps
    Criei uma extensão para embutir Google Maps em templates Jinja do Flask.

Quais sites, blogs ou listas de discussão que você mais leu neste ano?

Quais são as principais 3 coisas que você deseja aprender no próximo ano?

  • Uma nova linguagem. Talvez Go ou Rust (Ainda não decidi) e também desejo melhorar meus conhecimentos de Lua e quem sabe escrever algum projeto web com Lua.
  • Python 3.4 Aync Framework
  • RST (Preciso muito começar a escrever docs no sphinx)

Qual e o software, app ou biblioteca que você deseja que alguém crie no próximo ano?

  • Um micro framework para web com Lua (já conheço o kepler mas gostaria de algo mais parecido com Flask ou Bottle)
  • Um ORM para o OrientDB com suporte a Flask e Flask-Admin
  • Uma ferramenta gráfica para o MongoDB Aggregation Framework (quem sabe o Aqua Data Studio não implementa?)

Planos para 2014

  • Trabalhar menos e dormir bem
  • PythonHub
    pythonhub.com é um projeto de rede social para Pythonistas que já comecei a desenvolver em 2013, neste ano o projeto decola!
  • Mais Quokka CMS e melhorias em seus módulos
    Quokka deve se tornar um "CMS Framework" e já te issues abertas sobre isso no github, tambem precisa de melhor documentação e guias de deploy.
  • Escreverei mais no blog (sobre Python e Mongo)
  • Terminarei meu novo livro (já iniciei os rascunhos de um novo livro de desenvolvimento web com Python e Flask, é um guia para iniciantes e escrito em português!)
  • Novas turmas no cursodepython.com.br, muita coisa pronta para as novas turmas online de web2py, Flask e Mongo e o novo sistema de ensino.
  • Fechar minha empresa (não aguento mais pagar tanto imposto)
  • Ir ao MongoDB World Conference
  • Ir a Rottnest Island (Australia) conhecer um Quokka de verdade

E você?

quer participar? copie as peguntas faça uma postagem e tuite com a hashtag #2014pythonmeme.

  • Qual o framework, aplicação ou biblioteca mais legal que você descobriu neste ano?
  • Qual nova técnica de programação você aprendeu neste ano?
  • Qual projeto Open Source que você mais contribuiu neste ano? e o que você fez?
  • Quais sites, blogs ou listas de discussão que você mais leu neste ano?
  • Quais são as principais 3 coisas que você deseja aprender no próximo ano?
  • Qual e o software, app ou biblioteca que você deseja que alguém crie no próximo ano?
  • Planos para 2014