Closer To The Ideal

Chris Renner hates database abstraction layers, but I think he doesn’t know how to use them. He writes:

Propel actually does manage to provide some support for joins. I do not like Propel’s reliance on the “*” wildcard in the linked example, but perhaps that is just for simplicity in the documentation. Good SQL explicity names the columns to retrieve, and except for in-development testing, you should rarely if ever use “*” …its lazy coding and wastes resources (see Rudy Limeback’s Simply SQL for a more detailed argument against “*”). I’d like to see Propel, or any other ORM/DAL class deal, however, with a query such as this:

SELECT SQL_CALC_FOUND_ROWS
projects.id
, faculty
, author
, author2
, author3
, date_created
, users.last_name as faculty_last
, users.first_name as faculty_first
, users2.last_name as author_last
, users2.first_name as author_first
, dept.dept_name
, project_title
, project_type
, staff_no
, project_status
, organization.org_name
, project_date
FROM projects
LEFT OUTER JOIN users as faculty ON projects.faculty = users.id
LEFT OUTER JOIN users as users2 ON projects.author = users2.id
LEFT OUTER JOIN dept ON projects.dept = dept.id
LEFT OUTER JOIN organization ON project.org = organization.id
LEFT OUTER JOIN staff ON project.staff_no = staff.id
WHERE (users.last_name LIKE ‘%smith%’ OR users.first_name LIKE ‘%smith%’)
AND project.title LIKE ‘%Material Composition of Unobtanium%’
ORDER BY Items.Entry_Date DESC
LIMIT 0, 15

The above example is a sanitized (names anonymized to protect the innocent) version of a real query I use to return paginated results in one of our web apps. Some of the joins are included dynamically based on search criteria, others are always included. The WHERE clause is likewise generated dynamically, as well as the ORDER BY, which is used for column-based sorting.

Show me a DAL/ORM class that can deal with this example and I’ll be your best friend forever.

I do almost all my development work nowadays with Symfony, using Propel as the ORM, so I work with Propel a great deal. One trick for handling multiple joins is to use a database view. This is a trick that halfer mentioned to me on the Symfony forums.

Recently I’ve been working for a client who is taking an old system and porting it to Symfony. The old system had a lot of complicated JOINs. So, for instance, I take this:

SELECT
c.id, c.textid, c.title, c.title_link, c.content_date, c.issue_date,
c.protected AS protect, c.allow_comments, c.show_comments, c.content_hint,
c.author1, c.author2, c.author3, c.author4, c.author_str,
c.subtitle, c.abstract, c.direct_url,
c.imgpath1, c.imgpath2, c.imgcaption1, c.imgcaption2,
c.content_type, c.live_date,
c.reference_link_1, c.alt_info_1,
b.blog_name, b.blog_id,
cao_author_xref.author_id, cao_author_xref.state,
cao_author_xref.content_type AS content_t,
it.issue_text_date,
d.literal, d.alias, d.cdaobject, d.cdaaction, d.channel, d.adsite,
d.adzone, d.indexable
FROM cao_author_xref
LEFT OUTER JOIN cao_content c on (cao_author_xref.content_id = c.id
and cao_author_xref.state=c.state)
LEFT OUTER JOIN cao_issue it on c.issue_date = it.issue_date
LEFT OUTER JOIN cao_blogs b on c.reference_link_1 = b.blog_id
LEFT OUTER JOIN cao_content_type d on c.content_type=d.id

and I turn it into a view:

CREATE VIEW cao_paginated_content_by_author AS
SELECT
c.id, c.textid, c.title, c.title_link, c.content_date, c.issue_date,
c.protected AS protect, c.allow_comments, c.show_comments, c.content_hint,
c.author1, c.author2, c.author3, c.author4, c.author_str,
c.subtitle, c.abstract, c.direct_url,
c.imgpath1, c.imgpath2, c.imgcaption1, c.imgcaption2,
c.content_type, c.live_date,
c.reference_link_1, c.alt_info_1,
b.blog_name, b.blog_id,
cao_author_xref.author_id, cao_author_xref.state,
cao_author_xref.content_type AS content_t,
it.issue_text_date,
d.literal, d.alias, d.cdaobject, d.cdaaction, d.channel, d.adsite,
d.adzone, d.indexable
FROM cao_author_xref
LEFT OUTER JOIN cao_content c on (cao_author_xref.content_id = c.id
and cao_author_xref.state=c.state)
LEFT OUTER JOIN cao_issue it on c.issue_date = it.issue_date
LEFT OUTER JOIN cao_blogs b on c.reference_link_1 = b.blog_id
LEFT OUTER JOIN cao_content_type d on c.content_type=d.id

Now I add it to the config/schema.yml file as if it was a real database table:

cao_paginated_content_by_author:
id: ~
textid: { type: varchar(200) }
title: { type: varchar(200) }
title_link: { type: varchar(150) }
content_date: { type: timestamp }
issue_date: { type: date }
protect: { type: varchar(1) }
allow_comments: { type: varchar(1) }
show_comments: { type: varchar(1) }
content_hint: { type: varchar(1) }
author1: { type: integer }
author2: { type: integer }
author3: { type: integer }
author4: { type: integer }
author_str: { type: longvarchar }
subtitle: { type: longvarchar }
abstract:
peerName: cao_abstract
phpName: cao_abstract
type: LONGVARCHAR
required: true
direct_url: { type: varchar(250) }
imgpath1: { type: varchar(150) }
imgpath2: { type: varchar(150) }
imgcaption1: { type: longvarchar }
imgcaption2: { type: longvarchar }
content_type: { type: integer }
reference_link_1: { type: integer }
alt_info_1: { type: varchar(200) }
blog_name: { type: varchar(200) }
blog_id: { type: integer }
issue_text_date: { type: varchar(100) }
author_id: { type: integer }
state: { type: varchar(1) }
content_t: { type: integer }
live_date: { type: timestamp }
literal: { type: varchar(50) }
alias: { type: varchar(50) }
cdaobject: { type: varchar(50) }
cdaaction: { type: varchar(50) }
channel: { type: varchar(50) }
adsite: { type: varchar(50) }
adzone: { type: varchar(50) }
indexable: { type: varchar(1) }

Now I run the Symfony command that autogenerates my Propel classes based on the above schema.yml.

Now to run the above JOIN statement, using Propel, I only need a few lines of code. If I want all the rows from the JOIN where STATE equals “P” I write:

$c = new Criteria();
$c->add(CaoPaginatedContentByAuthorPeer::STATE, “P”);
$arrayOfItems = CaoPaginatedContentByAuthorPeer:doSelect($c);

That is 3 lines of code. And it keeps the SQL out of my code, which I think is a big plus.

Scott7 posts some mod_rewrite rules that he found useful when running Symfony on shared hosting. This looks like it could be useful to others, so I post it here.

A little logic, great motivation and bang i’ve done it.
Options +FollowSymLinks +ExecCGI

RewriteEngine On

# we skip all files in /web
RewriteCond %{REQUEST_URI} ^/web/
RewriteRule .* – [L]

# we rewrite all other files with .something to /web
RewriteCond %{REQUEST_URI} \..+$
RewriteCond %{REQUEST_URI} !\.html$
RewriteRule ^(.*)$ /web/$1 [L]

# !!! UNTESTED !!! ##################################
# we check if the .html version is in /web (caching)
RewriteRule ^$ /web/index.html [QSA]
RewriteRule ^([^.]+)$ /web/$1.html [QSA]
#####################################################

# no, so we redirect to our front web controller
RewriteRule ^(.*)$ /web/index.php [QSA,L]

# big crash from our front web controller
ErrorDocument 500 “

Application error

symfony application failed to start properly”

the files in /config are safe, images, controllers work nicely. I don’t fully get the caching part and how to test it. I hope someone can verify and help me publish this for good.

UPDATE:
You’ll also need this at the beginning of your /web/index.php:

/* according to .htaccess */
$_SERVER['SCRIPT_NAME'] = '/index.php';

define('SF_ROOT_DIR',...

Some of the more interesting things I write nowadays seem to get posted in discussions on LinkedIn. I repost here what I wrote in response to someone who wanted to know how big the Symfony community is.

Everything I’ve seen and heard suggests to me that the Symfony community is small and stagnant. Consider this graph.

Drupal and WordPress are both much, much larger than Symfony. More so, according to that graph, Symfony’s growth is limited compared to WordPress.

If you remove Drupal and WordPress, then it appears that Symfony is roughly the same size as Zend and CodeIgniter, and all of them have relatively flat growth. Consider this graph.

Of course, Google Trends is only one source of information, and only offers a rough approximation. It is important to check other data sources. So, for instance, we can double check the trend by looking at Indeed: consider this graph.

Here we see that the trend to Drupal is both much greater than Symfony, and also growing at a much faster rate.

This also interesting, comparing Symfony, Zend, Cake and CodeIgniter: consider this graph.

In this, the big winner is Cake, with Symfony coming in 2nd.

There is also the experience that Darren Hoyt and I have had trying to launch our 2 sites. Consider the high levels of activity on our WordPress site compared to the low levels of activity on our Symfony site:

http://www.wpquestions.com/

http://www.symfonyexperts.com/

Symfony Experts is newer, but we put far more effort and money into marketing it than we did with the WordPress site. Yet it still lags.

Also, consider that Symfonians.net is basically the official membership site for Symfony, yet it has less than 3,000 members:

http://symfonians.net/

It is like a rounding error when compared to Drupal.

Finally, there is my own experience on LinkedIn – I constantly get calls from headhunters who call me because I know Symfony, and always they say the same thing – “There are not many of you Symfony programmers, I’ve had a hard time finding people for the job I’m trying to fill.”

No matter what data source you consult, the conclusion is unmistakable: Symfony is a small community, and its rate of growth is slow.

Why is this? My own theory is that Symfony brings to PHP something that PHP has long been missing: a truly professional, industrial strength, object oriented architecture that allows PHP to be taken seriously in the enterprise. However, all of the people who would be truly interested in such a framework have long ago found a home using Ruby, Java or Python. In short, the kinds of people who gravitate to PHP are exactly the kind of people who are unwilling or unable to appreciate the beautiful elegance and power of a system like Symfony. The people who need something like Symfony gravitate to other languages.

Robert Speer has a great write up of Apostrophe:

Apostrophe is the easiest to use content management system (CMS) available to the open source community. An easy CMS means that content managers are more likely to use it, which means consumers will get better information and be more likely to follow the sites profit funnel.

For web solutions providers Apostrophe is a CMS solution that bypasses the commodity hell of Wordpress, Drupal, and Joomla by providing a unique value differentiation. Apostrophe also has the advantage of being built on an enterprise grade web framework used by sites like Delicious, Dailymotion, Yahoo! Answers, and Yahoo! Bookmarks. Symfony provides a consistent structure that encourages collaboration, and the large community of developers already familiar with Symfony mean help is available.

Worth knowing about: a listing of jobs for Symfony developers.

Here is a great post on extending a blog plugin for Symfony. There is a new hack attack going around for WordPress, so I’m thinking about switching to Symfony for my blogging.

A controversial claim, but he makes an interesting case. Russ argues that Symfony sets a high standard for all other open source projects to aspire to. No doubt this is true.

Over on Symfony Nerds, I wrote about Potencier’s ideas about a template engine. I just noticed that Eli White had also written an intelligent reply, well worth reading.

I’ve a new post up at Symfony Nerds. In it, I look at the implications of a RESTful architecture for Symfony. Should every module just have 4 actions, read, write, update and delete?

Scott Meves points me to this presentation by Kris Wallsmith. Lots of information about how to scale up Symfony, and a plugin is thrown in for handling multiple database connections. I intend to come back and dive into that subject more later on.

I agree with this: Netbeans is a good IDE for Symfony development. I think this is especially true if you, like me, also develop with JVM languages. This last year I was doing some Java/Swing and Groovy projects, so I started using NetBeans again. It has a plugin for doing Symfony work – very handy.

Colin Steele writes about the challenges that he and his team are facing at Hotelicopter:

We’ve officially run headlong into one of Ruby on Rails’ deficiencies: programming in the large. We’re not interested in computer science-y solutions, only pragmatic ones.

I’m curious what is excluded by the phrase “computer science-y solutions”? I would normally interpret that to mean “we are looking for well tested solutions with wide deployment” but elsewhere he writes:

We’re currently investigating a spectrum of new technologies in the NoSQL realm, including Tokyo Tyrant, MongoDB, Amazon’s SDB, CouchDB, Voldemort, and more. Tis’ a dizzying mix, and things are popping in the space.

So clearly they are looking at some cutting edge technologies. Colin links to an essay about Programming in the large which includes this:

Maintenance and locality are strong arguments in favor of immutability. The less aspects of an object can be changed, the less you have to worry about the execution history. If an object has a two-phase initialization sequence (e.g. this is so in C++ if you need a virtual function during initialization), you have to make sure that the objects are properly initialized; code that gets handed over such an object will have to check that it’s initialized (if only in an assert()). This all vanishes if the language makes sure that no object remains uninitialized, and doesn’t force a two-phase initialization on programmers like C++. (In C++, the “wrong” design decision was that objects mutate from the base type to the subtype when the various constructors are run. It’s this kind of far-reaching consequences (IOW non-locality) that makes language design an art.) If you take immutability to its extremes, nothing can ever be changed. If you wish to change the world, you write a function that returns a list of changes and let the run-time system inspect that list and execute the proper actions. If you have an interactive program, you emit a list that has a function pointer at its end; the function gets fed the next input and is expected to generation another action list.

In the last few years, many programmers have pointed to mutability as one of main problems they face when they build larger systems. Chas Emerick does a good job of highlighting the problem in his post “All my methods take 316 arguments, and I like it that way”

316 arguments to a method (which I don’t think is actually possible in the jvm, but bear with me)? “That’s absurd!”, you’d say. The problem, of course, is that the 3-arg doSomething actually has far more arguments than its signature implies:

The behaviour of every function in a mutable, imperative environment is dependent upon the state of all of the other (variables|attributes|bindings|whatever) in your program at the time the function is invoked.

So, if you have 313 other variables in your program, that 3-arg doSomething is functionally (ha!) operating over 316 arguments.

Would you ever intentionally write a method signature that takes 316 arguments? Would you use any library that contained such a function signature? No? Then why are you using tools that force such craziness upon you?

Chas says “The languages are ready” and he links to some of the major functional languages: Erlang, Clojure, F#, and Fantom. In comments, his readers add in their favorite functional languages: OCaml, Haskell, etc.

One of his readers challenges the idea that functional languages are safer than imperative languages by offering this:

Regarding functions changing state, what about things like this in clojure, Isn’t it like global variables in imperative lang?

(def state (ref #{}))

(defn function that updates state)

(defn another function that updates state)

Chas responds:

You bet. Clojure is not a purely functional programming language, so you can have as much shared state as you want – but the language is going to make you work for those bits of shared state, so you have to “pay” for them. Conversely, imperative languages like Java et al. make you work to achieve immutability, and provide nothing in the way of enabling persistent data structures, etc.

The point is that defaults matter, a lot.

I like the word “default” in this context. In his 2001 book, Effective Java, Joshua Bloch wrote “Favor immutable objects over mutable.” When writing a big system in Java, you work to make your system immutable. In a language like Clojure, the default is just the opposite – you work to make parts of your system mutable.

I have very little experience with functional programming. I am just learning Clojure now (Lisp redone for the JVM). I can not say what benefits its brings. I’m looking forward to learning more in this area. It’ll be interesting to see where functional programming comes to be regarded as a “best practice”. Certainly, it will be interesting to see if CTO’s start using these languages at startups, or whether they will be regarded as “computer science-y solutions”.

I’ve somewhat more experience with the web app frameworks that have emerged since 2004. I’m interested in what Colin wrote here:

I suspect as we muddle along we’ll develop a component-level (service level if you prefer) version of the Law of Demeter, which will drive us to make the right decisions for decoupling. I’m not too worried about that. However, we definitely have issues with reuse. Currently the Ruby on Rails state of the art solution for reuse is the gem. Which, let’s face it, is a pathetic solution.

Some of the frameworks seem to encourage bad habits. I’ve already written of Symfony’s weaknesses in Symfony versus The Law Of Demeter: does Symfony promote bad habits?.

When Ruby on Rails first emerged it was targeting web apps, not web services. Rails has a lot of imitators: Groovy/Grails, PHP/Symfony, etc. These all help create web sites, but not necessarily web services. I suspect a new generation of frameworks will be needed to make this kind of work easier:

The place we’re aiming for is a highly decoupled (and scalable), cohesive set of services, joined through REST APIs and/or fully reused common business models.

In their book Restful Web Services the authors Leonard Richardson and Sam Ruby talk about “the human web” and the “programmable web”. This is from page 2:

The Web you use is full of data: book information, opinions, prices, arrival times, messages, photographs, and miscellaneous junk. It’s full of services: search engines, online stores, weblogs, wikis, calculators, and games. Rather than installing all this data and all these programs on your own computer, you install one program – a web browser – and access the data and services through it.

The programmable web is just the same. The main difference is that instead of arranging its data in attractive HTML pages with banner ads and cute pastel logos, the programmable web usually serves stark, brutal XML documents. The programmable web is not necessarily for human consumption. Its data is intended as input to a software program that does something amazing.

Originally, frameworks like Rails were created to help speed the production of sites for the human web. They have evolved since then, Rails in particular. In fact, Richardson and Ruby use Rails for many of the examples they offer in the book, about how to correctly build a RESTful web service. And yet, the scaffolding systems in these frameworks still tend to automate the production of CRUD web pages, rather than PGPD services. (I do not know the state-of-the-art with Rails, so someone can tell me if I’m wrong about its scaffolding.)

Richardson and Ruby suggest that every module (resource) in a RESTful web service should expose just 4 actions:

POST
GET
PUT
DELETE

These are the HTTP verbs, and they roughly correspond to the standard CRUD actions, except that POST is used for both Create and Update, and PUT is used for uploading files:

Create/Update
Read
Upload
Delete

I suspect we need a new generation of frameworks, or at least new scaffolding systems for the existing frameworks, that automate the setup of PGPD services. That seems like the next obvious step forward.

[What follows is a comment I posted over at Hacker News.]

My impression is that there was a stretch when some combination of the public mood and the government’s emphasis conspired to encourage small startups. The 1980s and 1990s were clearly good in this respect. The mood of the last decade has been increasingly punitive. Sarbanes-Oxley is the most clear example of this. What once would have been treated as a civil matter is now treated as a criminal matter. Entrepreneurs are now faced with jail time instead of lawsuits. This can only have a chilling effect on innovation. I think it is urgent that everyone who cares about entrepreneurial culture in America to make the argument that innovation in business depends in part on tolerance, and that, in practical terms, this means most matters of conflict should be treated as civil rather than criminal cases.

A comparison might be made to the evolution of bankruptcy law. Before the mid 1800s, most Western countries treated bankruptcy as a criminal matter, rather than a civil one. The liberalization of bankruptcy law was one of the factors that allowed our modern economies to gain the dynamic nature they now enjoy. The public’s mood changed during the 1800s as it became more obvious that many times entrepreneurs failed with their first venture. They needed a second chance, when they were often more successful. John Bayer, who created what became Bayer aspirin, is an outstanding example of this – at first he tried to build a liquor business, but it failed. His father-in-law was suffering arthritis, and therefore drinking large amounts of willow bark tea – the only known source acetylsalicylic acid. John Bayer then put the willow bark tea through the distillery equipment he’d bought for his liquor business – and thus asprin was created. The point is, he needed a second chance to become successful. Many entrepreneurs are in this category.

Since this is Hacker News, I would guess that most of us know someone who has tried to do a startup, and failed on their first attempt. Many of us also know entrepreneurs who tried again, and met with greater success on successive tries. Tolerance of failure is the first pre-requisite of a dynamic economy.

More so, if you have any friends who have attempted to launch a startup, ask yourself under what circumstances you think your friends should go to jail.

I posted a similar comment some months ago, and I mentioned how many lives might be saved by the next wave of medically-focused startups. Someone responded:

“When you cross the line into experimenting with medical treatments, you’re not gambling with other people’s money, you’re gambling with lives. You can’t just equate it to any other kind of start up, it has to be held to a higher standard.”

I want to repeat, many, many industries can lead to people’s deaths. There is nothing unique about medical innovation. If you build a new kind of jet engine, which gets through testing but which then is responsible for a spectacular crash, then your product has killed a few hundred people. And yet, unless there was fraud in the documentation of the tests, there have not been criminal cases in the past. Right from its creation, decades ago, the FAA has taken a strong line against criminal – the feeling has always been that criminal prosecutions would stifle the free flow of information, and the only way to save lives over the long-term is through the free flow of information.

Many other fields can cause people to die – industrial automation, the transport and disposal of toxic chemicals, the construction of buildings (which could then fail and kill people). All industries are in need of innovation all of the time, yet innovation brings with it risk, including the risk of death. How much innovation will we get if we make these matters criminal?

I should emphasize, just in case people forget, that fraud has always been criminal. It has been criminal for centuries. So the move to criminalize more aspects of business is not a move to make fraud criminal. If you think that the Sarbanes-Oxley Act made fraud criminal, then you are mistaken. Fraud has always been criminal.

Sarbanes-Oxley is representative of the new trend. The overall goal was to encourage greater accuracy in the reporting of a company’s financial health. This goal could have been reached through a variety of methods, including both the carrot (rewards) and the stick (punishments). Rewards could have included tax breaks for meeting some additional level of compliance. Punishments could have included fines levied against companies that failed to meet a higher level of compliance. These approaches would not have raised the risk of jail time for CEO’s. Instead, Sarbanes-Oxley decided to go with the heaviest kind of punishment of all – to treat infractions as criminal offenses, potentially meriting jail time.

This punitive attitude is going to have a chilling effect on the amount of innovation we can expect in any field.

I’ve a new post up at Symfony Nerds. This looks at the evil of utility classes full of static methods, a vice that I’m guilty of on every project. I examine possible ways to refactor the code to get a healthier overall architecture.

Darren Hoyt and I have launched a new website. My ideas about this site grew out of m frustration with Experts Exchange. I’ve written about the evolution of my thoughts over on the WP Questions blog.

Darren has written a summary of everything that WP Questions is, and isn’t.

We used the Symfony framework to build this site. I’ve written a somewhat technical post, over on Symfony Nerds, explaining why this was a good choice for us.

The site took Darren and I about a month to develop. I thought about the right structure for the database during the last week of October, and we got to work in early November. We both took a month long break from doing work for clients, and pretty much just worked on this site, full time, for the month. Our focus was a minimalist but sufficient feature set, a clean design, and a database structure that will be flexible in the face of future changes (among other things, I was wary of too many foreign key relationships).

This last spring I was thinking about question and answer sites where money might be exchanged. On May 5th, I wrote this in a email:

“I’ve the easiest time imagining building out a site for a particular niche… Personally, I’d love a site devoted to programming, where I could put up a question for $20 or $30. Just this week I lost 4 hours tracking down an obscure conflict in the Javascript used on a site – one script was used to fix the PNG transparency problem in IE6, and another script, which preloaded large images used in rollovers, was triggering the first script too often, adding extra images to the page. I would have loved to pay $25, or even $50, to let some IE6 specialist tackle that one. I could imagine being a customer of such a site.”

Our first day went well:

1,389 visitors

200 users signed up

A few paid questions

150 tweets on Twitter

We are starting off focused on a small niche: web developers who use WordPress. If the site is successful, we will roll out similar sites for other, broader technologies: Java, Ruby, Rails, Grails, SQL, .NET, Oracle, etc.

As I said today on our blog, in reaction to user responses:

I’ve been working on websites for 10 years, and I’m very pleased with how our first day went – much better than average. We received a lot of valuable bug reports and thoughtful feedback. I was pleased to see some questions posted. Our fees are 9% plus 50 cents. PayPal takes about 3% and 30 cents, so our net profit from yesterday was less than $10 dollars, but, hey, that means we made more on our first day than Twitter made during its first 2 years. Which leaves me hopeful that we are, in fact, offering a service that provides real utility to people. And as we listen to your feedback and implement the better ideas, we hope to be able to offer an even greater service in the future.

Just saw this site referenced on the user mailist. Symfony-Check.org is a useful list of things to check when building a Symfony-backed website.

I’ve a new post up at Symfony Nerds, which looks at some of the bad coding practices which are common on some of the Symfony projects that I’ve worked on.

I’ve a new post up at Symfony Nerds. Fabien Potencier, the lead developer on Symfony, has recently introduced 2 new components related to templates in Symfony, including Twig, a template engine that will be introduced in Symfony 2.0. I try to offer an overview of the whole debate around these developments.

I’ll now be contributing to the Symfony Nerds blog. My first post is up, and it mostly reviews various bits of Symfony news and articles that I found interesting during the last week or so.

For people new to Symfony, this is a good write-up of the new forms framework, introduced a year ago in Symfony 1.1.

I am curious about Mobicules weblog. Some of their posts are self-serving, but they seem to be trying to get some real information out, which is more than what I’ve seen so far from most of the sites of Indian firms that I’ve looked at. How should I read this post on which platform to use for a massive social site that can scale to tens of millions of users?

ELGG: Due to a very flat and normalized database structure, the only way to scale horizontally with ELGG is to duplicate the database using MySQL replication on multiple servers. The negative with this approach is that it ends up duplicating the complete database, with the result that each machine running a copy of the replicated database will have to be very powerful server, rendering the solution expensive. (MySQL clusters is a standard way of doing it)

Drupal: Similar problems as ELGG with Drupal, but database is not as flat and normalized as ELGG. Scaling up in a similar manner as described for ELGG wity MySQL clusters would probably be cheaper with Drupal.

Symfony: Allows us to use a custom database structure. We can design the database and replicate it as we like. Symfony also uses its own query/object caching mechanism, which is efficient. As an example, the Yahoo bookmarks site supports 20 million users on Symfony.

…Our recommendation in a case where you are looking to scale up to 10s of millions of users would be write everything from scratch (and not use a ‘platform’ like ELGG/Drupal), that allows you to customize and tweak anything and everything. Symfony seems to have a good reputation for enabling creation of very large websites, and so, we would recommend using Symfony as a framework.

I actually agree with their conclusions.

I’m working on image gallery to be used on a site built with Symfony. So I’ve been reading up on all the different techniques out there for managing images for such a site.

Sitepoint has a good tutorial on how to use the sfThumbnail plugin:

We’ll use the sfThumbnail class provided by the plugin to create a thumbnail of each uploaded file. The best place to do this is in the setFilePath() method of the Photo class. Add the following to the Photo.php file:

// in lib/model/Photo.php
public function setFilePath($value)
{
parent::setFilePath($value);
$this->generateThumbnail($value);
}
public function generateThumbnail($value)
{
parent::setFilePath($value);
$uploadDir = sfConfig::get(’sf_upload_dir’);
$thumbnail = new sfThumbnail(150, 150);
$thumbnail->loadFile($uploadDir.’/’.$this->getFilePath());
$thumbnail->save($uploadDir.’/thumbnail/’.$this->getFilePath(), ‘image/png’);
}

One question I had to look up was how to restrict which files users are allowed to upload. That thread on the forum was useful.

Working with Symfony is the first time I’ve worked with Yaml files. As such, I still have to look up the info when I’m trying to do something like get an array out of a mass of Yaml configurations settings. Searching around, for info on the subject, I found the following, helpful conversation.

ifilipov writes:

Hey guys, my navigation is located in app.yml

all:
  navigation:
    Home: index.php
    About us: about.php
    Events: events.php
    Mailing List: mlist.php
    Contact Us: feedback.php

i’m trying to put all this in array, but it doesn’t work.

$nav = sfConfig::get(’app_navigation’);
print_r($nav);

halfer replies:

I believe to collapse things down to an array, you can also use the dotted syntax, something like this:

all:
   .settings
     navigation:
       Home: index.php
       About us: about.php
       Events: events.php
       Mailing List: mlist.php
       Contact Us: feedback.php

The dotted word can be anything, and is ignored by the YAML parser. Then to access it:

$nav = sfConfig::get(’app_navigation’);
print_r($nav);

Now things get strange.

I thought it would be useful to double-check with the official Yaml spec, to see what the dot really means. But the dot is not in use in any of the examples given in the spec. Instead,

To do a 2 dimensional array, there is this:

Example 2.3. Mapping Scalars to Sequences
(ball clubs in each league)

american:
   – Boston Red Sox
   – Detroit Tigers
   – New York Yankees
national:
   – New York Mets
   – Chicago Cubs
   – Atlanta Braves

Here is something I take to be what in PHP would be considered an associative array inside of non-associative array:

product:
   – sku : BL394D
      quantity : 4
      description : Basketball
      price : 450.00
   – sku : BL4438H
      quantity : 1
      description : Super Hoop
      price : 2392.00

But I find no examples using the dot notation. Which is odd.

(As to the conversation on the forum, partly I link to these articles to give them some rank with Google. I find that Google does a poor job of figuring out what pages are relevant to Symfony searches. I think this is partly because Symfony bloggers don’t link to many tutorials or quick tips, so Google doesn’t have many links on which to form a judgement about what is important.)

I’ve already mentioned one use of public_path(), which is to leave as much HTML as possible in the templates, so that designers have an easy time working with them.

This conversation was related, and caught my interest.

Mo Mughrabi wrote:

Generate absolute URL for upload directory

I was trying to generate an absolute URL inside my form classes using sfContext::getInstance()->getController()->genUrl(); but apparently genUrl only can be used when supplied a route or module/action but i cannot get it to return the absolute path of my uploads folder. Any idea how can i do that?

Eno responded:

We never found a good solution for this (we run across a cluster where we need absolute and cross-domain links). Now we are using the sfDomainRoutePlugin to give us this functionality in url_for() and link_to().

And then Mo Mughrabi said:

I found a way which helped me return the absolute url for any location i
need through the public_path helper.

First i had to include the helper through the getConfigurations

sfContext::getInstance()->getConfiguration()->loadHelpers(array(’Url’));

and then just use the public_path(’uploads/file.jpg’);

Reading this. I thought this was a funny commit note:

added more deprecated stuff

I mean, you can kind of guess the meaning, but the plain meaning suggests they are adding in code that is deprecated and should, therefore, be removed from the code.

Tom Boutell explains some features that are hidden in his pkToolkitPlugin plugin for Symfony:

* The pkHtml class. Many projects allow users to edit content via a rich text editor. And we all know what happens if the user pastes a Word document in there: the styles of the page wind up hopelessly munged. You can use striptags(), but that doesn’t clean up the CSS, so your page is still a mess.

A common workaround is to use FCK’s “paste as plaintext” mode, which thwarts attempts to paste rich text from another program. That works, after a fashion, but it’s frustrating for users. And none of the workarounds help if the user is actively trying to enter inappropriate HTML in a misguided attempt to re-style their site… then calling you to fix it for the 500th time.

HTML Tidy can do the job, but it has a reputation for being heavy and slow.

Or… you can just use pkHtml::simplify():

I am intrigued by Felix De Vliegher’s suggestion for a single filter that catches all of one’s exceptions in Symfony:

class ExceptionCatcherFilter extends sfFilter
{
public function execute($filterChain)
{
if (sfConfig::get(’app_exceptionCatcherFilter’)) {
try {
$filterChain->execute();
} catch (sfStopException $e) {
// This is an internally used symfony exception and shouldn’t be blocked
throw $e;
} catch (Exception $e) {
// Do something with the exception, other than just throwing it
}
} else {
$filterChain->execute();
}
}
}

Generally, I dislike this style of programming, where all errors are funneled through a central well. This style of programming is the one advocated by Anders Hejlsberg (inventor of the C# language):

There’s a bottom level exception handler around their message loop. That handler is just going to bring up a dialog that says what went wrong and continue. The programmers protect their code by writing try finally’s everywhere, so they’ll back out correctly if an exception occurs, but they’re not actually interested in handling the exceptions.

The throws clause, at least the way it’s implemented in Java, doesn’t necessarily force you to handle the exceptions, but if you don’t handle them, it forces you to acknowledge precisely which exceptions might pass through. It requires you to either catch declared exceptions or put them in your own throws clause. To work around this requirement, people do ridiculous things. For example, they decorate every method with, “throws Exception.” That just completely defeats the feature, and you just made the programmer write more gobbledy gunk. That doesn’t help anybody.

…It is funny how people think that the important thing about exceptions is handling them. That is not the important thing about exceptions. In a well-written application there’s a ratio of ten to one, in my opinion, of try finally to try catch. Or in C#, using statements, which are like try finally.

…In the finally, you protect yourself against the exceptions, but you don’t actually handle them. Error handling you put somewhere else. Surely in any kind of event-driven application like any kind of modern UI, you typically put an exception handler around your main message pump, and you just handle exceptions as they fall out that way. But you make sure you protect yourself all the way out by deallocating any resources you’ve grabbed, and so forth. You clean up after yourself, so you’re always in a consistent state. You don’t want a program where in 100 different places you handle exceptions and pop up error dialogs. What if you want to change the way you put up that dialog box? That’s just terrible. The exception handling should be centralized, and you should just protect yourself as the exceptions propagate out to the handler.

In contrast, I have always preferred the style advocated by James Gosling (inventor of the Java language), and the style preferred by Bill Venners, as articulated in this interview:

Bill Venners: I do catch exceptions in an outer loop sometimes. But most times my catch clauses tend to be spread around the program. I find that catch clauses usually have a natural home—the method that has enough contextual knowledge to know how to deal with the exception. How would you recommend people organize their catch clauses?

James Gosling: I tend to do catches much more frequently than Anders would have you do, because the knowledge of the situation is always fairly localized. When you try to open a file and it’s not there, you’re coping strategy is really determined by what you were going for. Some guy miles away isn’t going to know what to do. The code that tried to open the file knows what to do, whether it be trying a backup file, looking in a different directory, or asking the user for another filename.

Having one big catch clause on the outside really only works if your exception handling philosophy is simply to die. If you have an event loop, you can maybe cause that one event to just be tossed. If you have a plugin architecture, the enclosing environment could respond to a failure in the plugin by disabling it—like an app server deciding to disable a servlet if it sees failures. But if you’re not doing an event driven program or plugins, there isn’t an outside place where you can take big hunks of functionality and saw them off. On the other hand, typically you should have last ditch try catch blocks. If you’re writing a web server, for example, it’s a good thing to put a last ditch try catch block around processing a request. But pretty much all that a try catch block like that can do is blow the request away. There’s no ability to respond gracefully. There’s no ability to take account of local context to cope and adapt, which is really one of the key hallmarks of truly reliable software.

The key sentence is “Having one big catch clause on the outside really only works if your exception handling philosophy is simply to die.” Handling exceptions close to where they happen allows you to use variables that are within the current context to try to rescue the situation.

All the same, Felix De Vliegher’s idea is a wonderfully clean way to proceed with a Hejlsberg type of exception handling.

I previously mentioned that the worst thing about Symfony is how under-documented Propel is. I also notice that the situation as gotten better than it was in the spring of 2008, when I started using Symfony. There are more and more examples of how to do things, for instance, a Left Join:

$c = new Criteria();
$c->addJoin(PatientPeer::DOC_ID, DocPeer::ID, Criteria::LEFT_JOIN);
$this->patients = PatientPeer::doSelect($c);

All the same, in the future, I will probably use Doctrine. It is well documented.

Alvaro Videla thinks out loud about what design patterns are implemented in Symfony:

Some of the patterns involved in Symfony are:

* Front Controller
* Command
* Intercepting Filter
* Context Object
* Two Step View
* Helper Object or View Helper
* Table Data Gateway (i.e.: ArticlePeer.php)
* Row Data Gateway (i.e.: Article.php)
* Active Record
* Single Table Inheritance

This company developed their platform in the cloud, but eventually gave it up and moved to a dedicated server at RackSpace.

There are also technical complexities in that you have to provision EBS storage before you use it, and if you need to resize then you have to take a snapshot and rebuild from that.

We would have had to build our own infrastructure management system. This would have to handle the starting and failover of EC2 instances, backups and provisioning additional storage. In order to have no downtime when storage was being re-provisioned, we would need a second instance to replicate the database on. Aside from the extra server costs, all of this would have taken development time away from our efforts of improving the product itself.

We had the choice of working on infrastructure that makes no difference to the customer experience (but which would have been technically interesting and fun to develop) verses tangible progress with our product:

With six years of experience running my own software company I can tell you that nothing we have ever done at Fog Creek has increased our revenue more than releasing a new version with more features. Nothing. The flow to our bottom line from new versions with new features is absolutely undeniable.

- Joel Spolsky

And that’s the hidden cost – development time – something that is particularly important for a startup. As Jeff Atwood says, “Hardware is Cheap, Programmers are Expensive“.