Weapons of mass creation

Notes on Unicode Demystified at Pycon

Unicode In Python, Completely Demystified

[showing unicode error]

UnicodeDecodeError: 'ascii' code
...

Never seen this exception?
Seen it and sort of fixed it?
This is a confusing error, we’ll help u not get there.

What strings really are in Python

Why use Unicode in Python?

handle non-english
third party modules use unicode internally
safely accept arbitrary text
you will love Unicode
you will hate …

[Examples of you u might use text]

web page
database
output formats

Let’s open a UTF-8 file

just a string of bytes
encoded in utf-8 format ‘Ivan Krstixc4x87’
a c with a hachek takes two bytes

ASCII

UTF-8 is an extension of ASCII
c wiht hachek cannot be encoded in ASCII

s.decode(encoding)
s.encode()

Unicode is more accurate

u'Ivan Krsti\u0107'

a way to represetn text without bytes
unique number (code point for each character)

Unicode, the ideal

the abstract form of text
to save Unicode to disk you need to encode it

Unicode Transfromation Format

variable byte representation
first 128 chars encoded just like ASCII

Ian Albert’s Unicode chart

6 feet long

Decoding text into Unicode

it’s mostly automatic
happens alot in third party
python does it for you

Python Magic

Default encoding for Python 2 is ASCII

Decode early
Unicode everywhere
Encode Late

Shortcuts

codecs.open()

how to work with Unicode in Python 2

Glimpse at Unicode in Python 3

they fixed Unicode
str is a Unicode object
no more u’text’ syntax
default encoding is UTF-8
- woo!
Tries to guess the file encoding
- You will still need to declare encodings

Thanks Leapfrog Online

Ask lots of questions

String literals, how will it work?

Top of file you will need to specify encoding.

Notes on Django Tutorial at Pycon

Jacob Kaplan Moss

Querysets are cheap and queries (db hits) are expensive.

Models, Views, Urls and Templates, all you need to know to get started.

Views

Where you gather all the “stuff” that you want to display.

Smugmug exploit.

Your URL is a part of you application, you should design it. URLs are user interface. Some frameworks try to hide URLs, but URLs are part of the web and part of the UI.

URLs are “hackable”. You can edit a URL so you can move up a directory, etc. This used to be a “expert” feature, but nowadays people see a logic to it. People chop URLs to get there. Users are getting mor savvy.

Django doesn’t lock you in to a particular URL structure.

Object mapping

Unlike Rails which will by default follow your controllers and build URLs automatically.
Relationships defined
URIs should be opaque. They should define a resource but not what’s behind it.

ROOT_URLCONF is where Django starts.

import * is evil unless the package is designed for it.

from django.conf.urls.defaults import *
urlpatters = patterns('', ...)

Regular expressions, full re power is there, but you probably won’t need it.

Tips:

If you’re repeating yourself in patterns then you probably should move that out into a separate url file i.e. include(‘cheeserater.packages.urls’). Just don’t anchor the pattern with $.
You can get a little performance increase to move common URLs to top of patterns list.

Dissecting a request /documentation/url_dispatch/

GET /packages/Unipath/
ROOT_URLCONF
cheeserater.urls (walking down the patterns trying to match against the path)
(r’^packages/’,…)
include (‘cheeserater.packages.urls’)
…
call the view function with the first arg as request

A view function is a python function that takes a request object and returns a response object.

MVC differences

Your view is responsible for collecting and collating data but does not include presentation data.
Visual representation is in a template.

Using render_to_response() [view code example]

get_object_or_404(SomeModel, **kwargs)
  try:
        return SomeModel.objects.get(**kwargs)
  except SomeModel.DoesNotExist:
        raise Http404(...)

Templates

Glorified chunk of HTML with some variables.

PHP is a template language.
Ruby uses RHTML.
Python has dozens.
Django ships with one because we like it, you can use something else if you like. You can use a different one quite easily.

[showing example template]

Django template language has been designed for someone who is used to writing HTML. i.e. a Designer

The magic dot

p[“name”]
p.name
p.name()

You shouldn’t have to tell your template designers if something is a function or method … cheetah template engine idea.

Template filters: filter chaining with | built-in filters
Tags: usually do something with flow and logic, i.e. for, include, if then … same tags require end tags and others don’t

Template inheritance “big whoop idea / feature”

solves the problem of common code
base template
blocks / block tags
holes or places that will get filled in later
“the news hole” “i need three hundred words to fill this news hole”
putting a title block in the base template and let the child define it’s own if it wants to.

[block.super example]

Thought django template language was the stupidest idea when he first started at LJ world.
Most people are used to includes, while you can do that, it’s not designed for that.
[ inheritance example]
base template (absolute smallest piece of design, stuff that never changes)
might have base templates for each type of content.
requires {% extends %} to be right at the beginning.
typically 3 or 4 levels of inheritance

Why?

Lawrence world was first site built with Django.
- It was one template before they rebuilt it.
matching divs with includes sucks.
every designer he’s worked with thinks template inheritance is brilliant.

Inheritance Tips

{% extends %} must be the first thing in your template.
More {% block %}s are better. Throw empty blocks in your template.
- i.e. {% block extrahead %}
If you’re duplicating content then you’re missing a block.
{{ block.super }} keep it in mind
/documentation/templates/
/documentation/templates_python/

Croyden (postneo.com) friend

Wrote a good post about how LJ does mobile sites.

Generic Views

Saves time in doing common tasks.

What’s a generic view?

early on they realized same stuff was being repeating.
when copy and posting code, you know something is wrong.
only writing parts of views that differ.
pagination
- getting that right once and not thinking about it again.
what does a list of objects do?
- takes a request and a queryset. (and has lots of options)
generic views are just plain old python functions, just views.
- nothing that special about them other than just used alot and tested.
[view code example]
package_list(request, package_name):
- return list_detail.object_list(…)
[template code example]
include makes sense for pagination widget.

URLconf revisisted

(pattern, callback, extra)

Testing

Apology for being preachy.

“Tests are the Programmer’s stone, transmitting fear into boredom.” — Kent Beck

A good code base vs bade code base = quality of unit testing

Book: Test Driven Development (TDD)

“Never write any new code unless you have a failed test.” Drives Jacob crazy, TOO BORING.

“I didn’t do test driven developement. I do stupidity driven testing … I wait until I do something stupid, and then write tests to aoid doing it again.” — Titus Brown

“Never let one test break.” — Jacob

“Whatever happens, don’t let your test suite break, you’ll regret it …”

Testing Django

load sample data
deal with HTTP
setup and teardown
until recently was thought to be too hard
rail shipped with a test framework
- stubs out test for you when you start a new app
- testing isn’t THAT hard
inspired by rails except more pythonic

Doctest

Unit tests

Fixtures: loading data in a reliable manner

Test client

Email capture

allows email to be captured so you don’t actually send out email

Doctests

String inside triple quotes that looks like an interactive session.
Both tests and documentation.
Or create tests.py file.
/documentation/testing/
Siege (test requests/sec)

What else ya got?

ref: Report Lab (integrating PDF)

Notes on Django Code Lab

Django Code Lab
Jacob / James / Adrian

James = release manager
Practical Django Projects

Adrian = everyblock founder
Benevolant Dictators of Life
Def. Guide to Django

Jacob

Code Lab, part 1

Pim Van Heuvan, URL Design
Justin Lilly, unit testing
Richard House, model design
Peter Herndone, searching, optimizing
J Clifford Dyer, next/previous links

Part 2
Bob Haugen, preparing for deployment
Wiley Kestner, sginals
Eric St-Jean, REST APIs
Sean O Connor, order_by(”?”)
Honza Kral, many questions, many answers

Url Design
—————
Questions by Pim van Heuven, Belgium

The case of the URLconf monster
urls.py 2032 lines long
600+URL patterns
94 dicts of keyword arguments
generated dynamically

Worst-case matching
everytime u try to match a regex and fail, it takes time
worst case is a 404
600 regex matches before u get to 404

Break things up
They are already logically separated
Split into multiple files and use include with prefixes.
Drill down into smaller and smaller urlconfs.

Forms taking etra parameters.
Use cases
pass in an existing object to a form and derive
Getting Initial data
django.newforms.models.model_to_dict()
Passing extra parameters into a form
override init()
take extra arguments u want *args, **kwargs
call superclass init()
super() is important
Other Tricks
after super() call

Testing Django
——————
Justin Lilly

Unit tests

Model Design, Richard House
—————————————
“We just need one more field…”

EventType
EventTypeOption
Event
EventOption

Let’s build a form
pass in event_type

No more schema changes.

Peter Herndon
——————-
Searching: query optimization, executing raw SQL, and choosing the right tool for the job.
Works for research library.
synapse.mxkcc.org
synapse google code

The problem
/people/

Raw SQL

Django’s ORM is by far the weakest ORM available. There’s a certain point at which you will have to write SQL.

from django import db

sql = “SELECT… FROM… WHERE end = %s…”
params = [end_date,…]

cursor = db.connection.cursor()
cursor.execute(sql,params)
for row in coursor:
…

Read your db optimizer docs.

“Bad programmers worry about the code. Good programmers worry about data structures.”

Fulltext search has been done.
Google Coop, Lucene, Solar, Sphinx

Search Integration with Django
djapian, django-sphinx, djangosearch (developed at LJWorld)

from djangosearch.indexer import ModelIndex

results = Document.index.search(query)

Searching over a flat structure is faster than nested.

J. Clifford Dyer
————————
Handling Previous/Next Links

Dave Lowe
————-
When not to use the admin
Moving from Cold Fusion to Django

Apps don’t need to be part of the project.
Multiple sites that share apps/content
Use Site object to
Editor in Site A should not be allowed to Site B.

new forms admin will help that but for now just write your own views.

Get away from having applications inside a project.

Bob Haugen
——————
Prepping for deployment

Coding defensively.
serving media using builtin static server when DEBUG is true.
template dirs based on
urlresolvers.reverse() in a deployed site
raise Http404(“Invalid order ID”) shows up in debug mode

Develop and Deploy
manage.py runserver
Apache + mod_python or mod_wsgi

Pieces of the rest of the stack that
reverse proxy load balancer
Perlbal
(from livejournal)
Apache mod_proxy

spoonfeeding (reverse proxying)
mod_python is moving fast but webserver is slowly moving bits.
load is 0 but new requests are waiting on IO
allow perlbal to handle trickling of data
abstraction of hardware resources
add and remove servers without downtime
perlbal docs kind of suck/non-existent.
ask questions on the list.

Use Memcached
livejournal

Wiley Kestner
——————-
Signals

Notifications

User subscribes to object
When this object changes i want different ways of being notified
User chooses method of notification

Design Pattern (Observer
dispatcher, django has one built in

Dispatcher 101
some piece of code sends a “signal”
just a predefined python obj
django defines some useful signals
django.dispatch.dispatcher.send()

def save_callback():
print “Something just go saved”

dispatcher.connect(save_callback, signal=…)

need to guarantee it gets excecuted
put in models.py

Let’s write a simple listener
…
No documentation yet (Jacob’s fault, he’ll finish it during the sprint)

http://toys.jacobian.org/presentations/