Mastering Google App Engine

Mohsin Shafique Hijazee

• Publisher: Packt Publishing
• Year: 2015
• eBook ISBN: 9781784396671
• Edition: 1
• Topic: Computer Science
• Subtopic: Cloud Computing
• Index: Computer Science

Build robust and highly scalable web applications with Google App Engine

About This Book

• Get an in-depth look at how Google App Engine works under the hood
• Design and model your application around Google's highly scalable distributed NoSQL datastore to unlock its full potential
• A comprehensive guide to ensure your mastery of Google App Engine

Who This Book Is For

If you have been developing web applications in Python or any other dynamic language but have always wondered how to write highly scalable web applications without getting into system administration and other plumbing, then this is the book for you. No experience in writing scalable applications is required.

What You Will Learn

• Scale and develop your applications with Google App Engine's runtime environment
• Get to grips with request handling mechanism and write request handlers
• Deep dive into Google's distributed NoSQL and highly scalable datastore and design your application around it
• Implement powerful search with scalable datastore
• Perform long-running tasks in the background using task queues
• Write compartmentalized apps using multi tenancy, memcache, and other Google App Engine runtime services
• Handle web requests using the CGI, WSGI, and multi-threaded configurations
• Deploy, tweak, and manage apps in production on Google App Engine

In Detail

Developing web applications that serve millions of users is no easy task, as it involves a number of configurations and administrative tasks for the underlying software and hardware stack. This whole configuration requires not only expertise, but also a fair amount of time as well. Time that could have been spent on actual application functionality.

Google App Engine allows you develop highly scalable web applications or backends for mobile applications without worrying about the system administration plumbing or hardware provisioning issues. Just focus writing on your business logic, the meat of the application, and let Google's powerful infrastructure scale it to thousands of requests per second and millions of users without any effort on your part.

This book takes you from explaining how scalable applications work to designing and developing robust scalable web applications of your own, utilizing services available on Google App Engine.

Starting with a walkthrough of scalability is and how scalable web applications work, this book introduces you to the environment under which your applications exist on Google App Engine. Next, you will learn about Google's datastore, which is a massively scalable distributed NoSQL solution built on top of BigTable. You will examine the BigTable concepts and operations in detail and reveal how it is used to build Google datastore. Armed with this knowledge, you will then advance towards how to best model your data and query that along with transactions. To augment the powerful distributed dataset, you will deep dive into search functionality offered on Google App Engine. With the search and storage sorted out, you will get a look into performing long running tasks in the background using Google App Engine task queues along with sending and receiving emails. You will also examine the memcache to boost web application performance, image processing for common image manipulation tasks. You will then explore uploading, storing, and serving large files using Blobstore and Cloud storage.

Finally, you will be presented with the deployment and monitoring of your applications in production along with a detailed look at dividing applications into different working modules.

Style and approach

This book is an in-depth guide where you will examine the problems in the context of highly scalable web applications. This book will take you through the libraries, services, and required configuration and finally puts everything together into a small web application that showcases all the capabilities of Google App Engine.

Mastering Google App Engine

---

Table of Contents

Mastering Google App Engine

Credits

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Understanding the Runtime Environment

The overall architecture

The challenge of scale

How to scale with the scale?

Scaling in practice

Infrastructure as a Service

Platform as a Service

Containers

How does App Engine scales?

Available runtimes

Python

The Java runtime environment

Go

PHP

The structure of an application

The available services

Datastore

Google Cloud SQL

The Blobstore

Memcache

Scheduled Tasks

Queues Tasks

MapReduce

Mail

XMPP

Channels

Users

OAuth

Writing and deploying a simple application

Installing an SDK on Linux

Installing an SDK on Mac

Installing an SDK on Windows

Writing a simple app

Deploying

Summary

2. Handling Web Requests

Request handling

The CGI program

Streams and environment variables

CGI and Google App Engine

WSGI

Problems with CGI

Solutions

What WSGI looks like

WSGI – Multithreading considerations

WSGI in Google App Engine

Request handling in App Engine

Rendering templates

Serving static resources

Cache, headers, and mime types

Serving files

Using web frameworks

Built-in frameworks

Using external frameworks

Using Bottle

Summary

3. Understanding the Datastore

The BigTable

The data model

How is data stored?

The physical storage

Some limitations

Random writes and deletion

Operations on BigTable

Reading

Writing

Deleting

Updating

Scanning a range

Selecting a key

BigTable – a hands-on approach

Scaling BigTable to BigData

The datastore thyself

Supporting queries

Data as stored in BigTable

The implementation details

Summary

4. Modeling Your Data

The data modeling language

Keys and internal storage

The application ID

Namespaces

The Kind

The ID

Pre-allocating numeric IDs

The string ID

The key

Modeling your data

The first approach – storing a reference as a property

The second approach – a category within a key

Properties

The required option

The default option

The repeated option

The choices options

The indexed option

The validator option

The available properties

Structured Properties

The computed properties

The model

The constructor

Class methods

The allocate_ids() method

The get_by_id() method

The get_or_insert() method

The query() method

The instance methods

The populate() method

The put() method

The to_dict() method

Asynchronous versions

Model hooks

Summary

5. Queries, Indexes, and Transactions

Querying your data

Queries under the hood

Single-property queries

Examples of single-property queries

Multiple property indexes

Working with indexes

The query API

The Query object

App

Namespace

Kind

The ancestor

The projection

Filters

The orders

Further query options

The keys_only function

The projection

The offset

The limit

The batch_size entities

The prefetch_size entities

The produce_cursors entities

The start_cursor entities

The end_cursor entities

Filtering entities

Filtering repeated properties

Filtering structured properties

The AND and OR operations

Iterating over the results

Conclusions

Transactions

Summary

6. Integrating Search

Background

The underlying principle

Indexing your data

Sample data

Indexing thyself

Documents

Fields

The text fields

Placing the document in an index

Getting a document

Updating documents

Deleting documents

Indexing the documents

Queries

Simple queries

Multiple value queries

Logical operations

Being specific with fields

Operators on NumberField

Operators on DateField

Operations on AtomField

Operations on TextField and HTMLField

Operations on GeoField

Putting it all together

Selecting fields and calculated fields

Sorting

Pagination

Offset-based pagination

Cursor-based pagination

Facets

Indexing facets

Fetching facets

Asking facets via automatic discovery

Asking specific facets

Asking facets with specific values

Asking facets in specific ranges

Filtering by facets

Summary

7. Using Task Queues

The need to queue things

The queue

Defining queues

Adding to a queue

Processing tasks

Putting it all together

Using a deferred library

Pull queues

Summary

8. Reaching out, Sending E-mails

About e-mails

Sending e-mails

The object-oriented API

E-mail on the development console

Headers

Receiving e-mails

Handling bounce notifications

Putting it all together

Summary

9. Working with the Google App Engine Services

Memcache

The Memcache operations

Memcache in Google App Engine

The Memcache client

The object-oriented client

Multi-tenancy

Automatically setting the namespace

The API-specific notes

The Datastore

Memcache

Task queues

Search

Blobstore

Blobs

Uploads

Getting BlobInfo

More BlobInfo methods

Serving

Reading

Users

Storing users in datastore

Images

Putting it all together

Summary

10. Application Deployment

Deployment configurations

Deployment revisited

Versions

The instance classes

Instance addressability

Scaling types

Manual scaling

Basic scaling

Automatic scaling

Modules

Accessing the modules

The dispatch.yaml file

Scheduled tasks

The Scheduled tasks format

Protecting cron handling URLs

Logs

The Remote API

AppStats

Summary

Index

Mastering Google App Engine

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: September 2015

Production reference: 1011015

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78439-667-1

www.packtpub.com

Credits

Author

Mohsin Shafique Hijazee

Reviewers

Aristides Villarreal Bravo

Johann du Toit

Acquisition Editor

Nikhil Karkal

Content Development Editor

Athira Laji

Technical Editor

Naveenkumar Jain

Copy Editor

Ting Baker

Vedangi Narvekar

Project Coordinator

Harshal Ved

Proofreader

Safis Editing

Indexer

Priya Sane

Production Coordinator

Komal Ramchandani

Cover Work

Komal Ramchandani

About the Author

Mohsin Shafique Hijazee started his programming adventure by teaching himself C, and later C++, mostly with the Win 32 API and MFC. Later, he worked with Visual Basic to develop an invoicing application for local distributors. In the meantime, .NET came a...

Table of contents

• Mastering Google App Engine