Closer To The Ideal

Whimsley points out that the Internet (and recommendation systems, in particular) expand choices for individuals, but potentially lower the overall range of things that are likely (emphasis on the word “likely”) to be experienced. The focus is especially on recommendation algorithms, such as the one on Amazon.com:

You can see that [in the graph above] on the left, in Internet World, a few products were chosen a lot, especially the one centred on about (-0.2, -0.2). In Offline World there are many more medium-sized dots, showing that the consumption of products is more equal. In Internet World one product has “gone viral” and gets chosen over 1500 times out of the total of 3600, while 26 products languish in the obscurity of being sampled fewer than ten times. In Offline World no single product is chosen more than 10% of the time, and only 14 products are sampled fewer than ten times. In short, niche products do better in Offline World than in Internet World.

While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.

The point here is similar to the point Clay Shirky made back in 2003:

A persistent theme among people writing about the social aspects of weblogging is to note (and usually lament) the rise of an A-list, a small set of webloggers who account for a majority of the traffic in the weblog world. This complaint follows a common pattern we’ve seen with MUDs, BBSes, and online communities like Echo and the WELL. A new social system starts, and seems delightfully free of the elitism and cliquishness of the existing systems. Then, as the new system grows, problems of scale set in. Not everyone can participate in every conversation. Not everyone gets to be heard. Some core group seems more connected than the rest of us, and so on.

Prior to recent theoretical work on social networks, the usual explanations invoked individual behaviors: some members of the community had sold out, the spirit of the early days was being diluted by the newcomers, et cetera. We now know that these explanations are wrong, or at least beside the point. What matters is this: Diversity plus freedom of choice creates inequality, and the greater the diversity, the more extreme the inequality.

Shirky is writing about weblogs, but what he says applies to books, CDs, videos – anything where people can influence the choices that other people make. Shirky’s point applies to the point that Whimsley is making about recommendation engines. This part applies equally to webloggers or videos:

We also know that as the number of options rise, the curve becomes more extreme. This is a counter-intuitive finding – most of us would expect a rising number of choices to flatten the curve, but in fact, increasing the size of the system increases the gap between the #1 spot and the median spot.

Shirky offered a hypothetical example of how this works:

To see how freedom of choice could create such unequal distributions, consider a hypothetical population of a thousand people, each picking their 10 favorite blogs. One way to model such a system is simply to assume that each person has an equal chance of liking each blog. This distribution would be basically flat – most blogs will have the same number of people listing it as a favorite. A few blogs will be more popular than average and a few less, of course, but that will be statistical noise. The bulk of the blogs will be of average popularity, and the highs and lows will not be too far different from this average. In this model, neither the quality of the writing nor other people’s choices have any effect; there are no shared tastes, no preferred genres, no effects from marketing or recommendations from friends.

But people’s choices do affect one another. If we assume that any blog chosen by one user is more likely, by even a fractional amount, to be chosen by another user, the system changes dramatically. Alice, the first user, chooses her blogs unaffected by anyone else, but Bob has a slightly higher chance of liking Alice’s blogs than the others. When Bob is done, any blog that both he and Alice like has a higher chance of being picked by Carmen, and so on, with a small number of blogs becoming increasingly likely to be chosen in the future because they were chosen in the past.

Think of this positive feedback as a preference premium. The system assumes that later users come into an environment shaped by earlier users; the thousand-and-first user will not be selecting blogs at random, but will rather be affected, even if unconsciously, by the preference premiums built up in the system previously.

Over the years, I’ve learned a great deal about programming from Artima, in general, and Bill Venners in particular. I was aware, circa 2003, that he was a big fan of Java. It is interesting to see him now decide to use a new language, and to compare Scala to the dynamic languages, as part of that process:

One of the most commonly cited productivity benefits of dynamic languages is that they enable much more concise programs compared to Java. While this is true, the productivity benefit here is actually conciseness, and that doesn’t require dynamic typing. I show how type inference in Scala lets you get much of the conciseness of say, a Ruby or Python program, but in a statically typed language.

Another productivity benefit of dynamic languages is that, via a feature called open classes in Ruby or Python, you can add a method to a class and then call it. You can’t do this in a static language. Nevertheless, I point out that the productivity benefit isn’t actually in the dynamic adding of the method. The benefit, instead, is in being able to call the method. And that you can do in static languages using various other techniques. In C# 3.0, for example, you can make it possible to call new methods on classes via C#’s “extension methods” feature. In The Feel of Scala talk, I show how you can use Scala’s implicit conversion feature to obtain that same productivity benefit.

Another oft touted productivity benefit of dynamic languages is that duck typing allows you to pass classes with common structures but no common supertypes to a method, which treats them uniformly. I point out once again that the productivity benefit isn’t actually the duck typing per se, but the ability to pass the different types. I demonstrate three ways to achieve that in Scala. First, I show “view bounds,” which is a completely static approach that involves implicit conversions. I also show structural typing in Scala, which is dynamic in the sense that reflection is used to invoke methods at runtime, but static in the sense that everything is checked at compile time. Lastly, I demonstrate using plain-old duck typing, exactly the same dynamic technique used in Ruby and Python, but this time in a static language. I show that as with duck typing in a dynamic language, spelling errors result in runtime errors, but then demonstrate that with a compiler plug-in, you can move that run-time duck typing error to a compile-time error, effectively extending Scala’s static type system.

Lastly, another commonly cited benefit of at least some dynamic languages is that their syntax is more flexible, allowing you to make domain specific languages. While this is also true, the benefit comes from the flexible syntax, not the dynamic typing. During the talk I show that dynamic typing is not required for syntactic flexibility. I demonstrate the flexibility of the Scala language, showing how you can define new control constructs and make internal DSLs.

To get a sense of how much Venners has changed, consider what he said to Matz when they spoke about Ruby:

In Ruby, I can add methods and variables to objects at runtime. I have certainly added methods and variables to classes, which become objects at runtime, in other languages. But in Java, for example, once a class is loaded or an object is instantiated, its interface stays the same. Allowing the interface to change at runtime seems a bit scary to me.

Some of this stuff used to strike him as scary. Like a lot of Java folks, Venners has become more open to the new methodologies advanced in recent years by the light weight dynamic languages.

We now live in a world where a corporation will report profits most years for a 10 year period, and then suddenly announce that, without massive subsidies from the government, it must declare bankruptcy.

One definition that I’ve heard used for “profits” is that it is “surplus cash, left over after all costs and investments of a business have been taken care of, unneeded by the enterprise and therefore free to be given back to the shareholders/owners.” That is, this is wealth that can safely be extracted from the enterprise and given to the owners. This definition is close to the one that Peter Drucker uses in his 1985 book, Innovation and Entrepreneurship.

But how can any business ever know if a given a dollar is unneeded? What if a great crisis lurks in the future? What if the money that a business gives back to shareholders this year turns out to be exactly the money that the business needs to survive the terrible financial crisis that just happens to break out next year? The overwhelming majority of all businesses that have ever been created have disappeared. How many businesses are there still surviving from the 1700s? If a business had the goal of surviving for as long as possible, surely, it would never declare a profit, never give any money back to shareholders. It would hoard every penny, and save it up for future disasters.

The question “How much profit does this company have?” reminds me of some aspects of the question “How long is the coast of Britain?” Obviously you can come up with some methodology which, if applied consistently, will give you an estimate of the length of the British coast. A different methodology will give you a different answer. Neither answer is more true than the other. There is no true answer, there is only the answer that a given methodology offers. And the same applies when you go seeking the profits of  a firm. You can use GAAP to estimate the profits of a firm. (Though, of course, GAAP is easy to game, as any of thousands of recent news articles will remind us.) You can use some other methodology, and it will give a different estimate of the profits of some particular firm. But it won’t give the truth.

If there is a truth to be discovered here, it would be that, over the very long term, there are no profits. There is no true surplus that a company can freely part with. There is no money that a firm won’t at least potentially need when the next crisis strikes.

But why would investors want to start a company if they can never get any of their money back? After all, most would argue that the purpose of a company is to generate profits. Most of the time, most people would argue that longevity should not be a goal for a business, but rather, longevity should simply be a side effect of being profitable. However, during the current crisis, both politicians and a large part of the  public seem to feel that longevity is more important than profitability. How else to explain the billions of subsidies being given to auto makers who have been unable to turn reliable profits for years?

As a practical matter, we might agree that there has to be some agreed upon methodology that gives an estimate of how much surplus wealth can reasonably be extracted from a business and given back to the shareholders. But it should be remembered that every dollar that is taken out of the firm, even in the most profitable years, is moving that company one small step closer to its death.

Clay Shirky is brilliant:

Revolutions create a curious inversion of perception. In ordinary times, people who do no more than describe the world around them are seen as pragmatists, while those who imagine fabulous alternative futures are viewed as radicals. The last couple of decades haven’t been ordinary, however. Inside the papers, the pragmatists were the ones simply looking out the window and noticing that the real world was increasingly resembling the unthinkable scenario. These people were treated as if they were barking mad. Meanwhile the people spinning visions of popular walled gardens and enthusiastic micropayment adoption, visions unsupported by reality, were regarded not as charlatans but saviors.

When reality is labeled unthinkable, it creates a kind of sickness in an industry. Leadership becomes faith-based, while employees who have the temerity to suggest that what seems to be happening is in fact happening are herded into Innovation Departments, where they can be ignored en masse. This shunting aside of the realists in favor of the fabulists has different effects on different industries at different times. One of the effects on the newspapers is that many of their most passionate defenders are unable, even now, to plan for a world in which the industry they knew is visibly going away.

The same thing has been happening in the American auto industry.

Marina Hyde is curious about the new era of celebrity, in which entertainers attempt to negotiate peace agreements in the Israeli – Palestinian conflict (as Richard Gere did). I appreciated this bit about Angelina Jolie’s tatoos:

Yet it seems she will at least endeavour to fill up Brad Pitt’s defaceable torso first. The first Brad unveiled was a forearm tattoo of Otzi the iceman. The second was a mysterious series of parallel lines that were diversely interpreted as a tribute to the great Nintendo platform games of the 80s, and a diagram of the New Orleans levee system. As it turned out, the speculation was way off target: Angelina herself had created the cryptic hieroglyph. “We went to Davos,” she said. “One night we didn’t have anything to do, so I was drawing on his back. It’s meaningful in that it’s us making angles and shapes out of each other’s body, that kind of a thing.” No. That is not why it is meaningful. It is meaningful because the kind of people who get so bored that they doodle on each other and turn the doodles into permanent tattoos are now attending the World Economic Forum. 

I disagree, the meaningful bit, in the sense of “What is changing?” is that we are now hearing these stories. Go back 20 years and people who went to Davos probably also had moments alone with their lovers. But back then, such information wasn’t shared publicly. I don’t see anything wrong with this, but this is one of those anecdotes that makes clear how society’s (international society, in this case) mores are changing.

David Van Couvering is sympathetic to my frustration regarding Sun:

 I remember when Rich Green did the Big Splash announcement for Java FX at Java One two years ago - I just groaned.  It was basically an “if you announce it they will come” strategy as far as I could tell – there was no there there.  The technology for the most part had not been built.  It was an announcement of an unrealized vision, not an actual working product.  And meanwhile there were not one but two competing solutions – Flex and Silverlight – from two of the best consumer software companies in the business.  Things did not look good – and still don’t.What made this whole Java FX thing really grating to me was that we as a company had to pull out all stops to get this thing to actually happen, since our VP of all software had put his reputation on the line by announcing it.  They pulled the best and the brightest from the NetBeans and Java Swing teams to work on Java FX.  They pulled our UI design resources.  They pulled our QA resources. And the now NetBeans was working on life support, and JavaFX struggled along, and continues to struggle along.   

Sun in 2009 reminds me of Apple in 1997, before Steve Jobs came back and rescued the company.  Great technology, great people, terrible leadership. I’d be pleased if there was someone else out there who could rescue the company. Sad to say, at the current moment, it looks like the company is going to disappear into IBM.

This is a very useful Javascript resource:

A lot of people are familiar with PHP’s functions, and though Javascript functions are often quite similar, some functions may be missing or addressed differently. The Javascript implementations should be as compliant with the PHP versions as possible, a good indication is that the PHP function manual could also apply to the Javascript version.

Porting crucial PHP functions to Javascript can be fun & useful. Currently some PHP functions have been added, but readers are encouraged to contribute and improve functions by adding comments. Eventually the goal is to save all the functions in one php.js file and make it publicly available for your coding pleasure.

In particular, PHP has a large number of functions for handling arrays. It will be convenient to have Javascript equivalents for them.

So, late last year I started learning JavaFX. I thought this would be a great way to learn about other areas of computer programming (since I mostly do PHP programming for websites). Sun Microsystems seems committed to pushing JavaFX as the next Big Thing. Sun has pulled their best people off of other important projects so that it could maximize its investment in JavaFX:

For almost two years now, Sun has been promoting JavaFX, the centerpiece of its vision for Java as a viable rich-client technology, and an alternative to Adobe’s Flex and Microsoft’s Silverlight. At heart, JavaFX is a JVM-based domain language for constructing rich user interfaces, as well as a set of APIs for that language.

…In spite of a thriving Swing community, and despite Swing’s large user base, Sun has re-focused its efforts around JavaFX over the past year-and-a-half, at the expense of Swing development. The most visible aspects of that change in focus is that many of the most experienced Swing developers left the company, such as Chet Haase (see Artima’s interview with Chet Haase), Hans Muller, or Scott Violet. The important Swing-related JSRs have also been stale for a long time now: the latest JSR 295 and 296 updates occurred in June, 2006, according to the JCP’s Web site.

For all the money that Sun is throwing at JavaFX, I’m surprised at how slowly it is finding a market.

I had assumed that JavaFX might make some progress in two areas:

1.) Online games.

2.) Cell phones.

Of course, online games was always a long shot, since Flash is so deeply entrenched. I was just looking at the games over at Casual Collective, they are all built with Flash. Online game developers are comfortable with Flash, and probably nothing can change that situation in the medium term.

Then there were cell phones. Here I thought JavaFX would make big progress. Jonathan Schwartz emphasized this in his talks at JavaOne. It’s been almost a year now, yet the press still doesn’t mention JavaFX when they talk about cell phones:

Heavyweights in the field include Nokia   and its Symbian operating system, BlackBerry maker Research In Motion, upstart Apple  and a staggering Microsoft  with its Windows Mobile system.

Each bring a special approach to the fight. Nokia has the unmatched size and reach, holding 40% of the smartphone market. RIM’s knockout punch is email. Apple’s iPhone is a showboat that dazzles crowds with media prowess. And Microsoft is fighting to carry an enormous Windows franchise to smartphones.

…Google funded Android, a smartphone operating system that HTC fashioned into a touchscreen phone called G1 sold exclusively by Deutsche Telekom’s   T-Mobile.  And while other phone makers, notably Motorola, have selected Android as its software partner, a new crop of G-phones has yet to arrive.

Meanwhile, Apple continues to advance its platform:

Apple unveiled new software for the iPhone that will support some long-anticipated features, such as copy-and-paste of text and picture messaging, as the company pushes to stay competitive in the phone market.

Apple also gave its vast network of software developers a slew of new options for upcoming applications, such as support for subscription models and automatic alerts, a move applauded by analysts.

“They’ve taken a few more steps ahead of the pack in the race,” said CCS Insight analyst John Jackson, adding that, in spite of the omission of certain features until now, the iPhone was still the most high-profile cell phone.

“Two years on they still have the cool phone and business model that everybody’s talking about and trying to emulate.”

Last year the folks at Sun were pushing the idea that JavaFX was the natural choice for Android. That might happen eventually, but for now, few developers have bought into Sun’s vision. Search for “Android JavaFX” and all you find is propaganda from Sun and some developers griping – you don’t find a single site singing about this idea with enthusiasm.  Search for “JavaFX” and all of the news is for developers – there are still no mentions of any consumer products. Sun seems to be running behind the rest of the pack. Most of the news articles focus on how amazing JavaFX will be when it is finally all done, and “all done” seems to always be a few more releases in the future:

However, some notes casually buried away in the 1.1 release notes suggest otherwise: the javafx.ext.Swing package isn’t available in the common profile so won’t work in mobile applications – this means no standard desktop UI components such as buttons, trees and listboxes. You do get one component, javafx.scene.control.TextBox. But the richness of the desktop component set just isn’t there in the mobile configuration. When you look at the existing mobile demos, you’ll quickly realize the colorful, motion-blurred, willy waving – such as a custom image masquerading as a button – divert attention from the lack of GUI components.

Yes, we’ve known all along that you need to build to the Common profile if you want to run on a mobile device; but if easy-to-use UI components aren’t part of that API, doesn’t that defeat the whole purpose of the very UI-centric JavaFX? More UI controls will be hitting the Common profile before JavaOne 2009, but until then mobile developers might find things a little half-baked. The Register does point out one saving grace: unlike the situation on the desktop, JavaFX mobile does not particularly lag offerings from Adobe (whose Flash Light isn’t in wide use) or Microsoft (which doesn’t even have a real mobile version of Silverlight yet).

It’s easy to want to root for Sun. They’ve been good to the open source community. They’ve donated billions of dollars worth of effort and software (Open Office, Java, etc). A world in which they are thriving seems likely to be a better than one in which Microsoft or Adobe are winning with their proprietary systems. Yet Sun doesn’t seem to have any understanding of how to succeed with consumer level technologies.

I’m still hopeful about JavaFX, mostly because I have faith in Google, and Android will allow Java to run on cell phones. But, damn, I am surprised about how slowly this technology is reaching the market (after all, Sun first announced JavaFX at the JavaOne conference back in 2007).

For much of the last 3 weeks I’ve used Windows laptop running XP (instead of my usual Ubuntu Linux machine) which gave me a chance to try out some of the browsers that have no Linux version.   One of the greatest aspects of the current web development scene is that most of the surviving browser projects are competing on how well they can implement web standards and HTML5 (contrast the current scene with 1996, when Microsoft set out to break the web, with its “embrace, extend and exterminate” strategy).

I’ve already noted that FireFox seems to have surpassed Internet Explorer, in terms of usage. Microsoft will hopefully soon kill Internet Explorer and replace it with something else.

I admire much of what Brendan Eich has done. And for the last several years, FireFox has been my web browser of choice. But FireFox drives me crazy with its demand for resources. Compared to any other browser I’ve tried, it demands more RAM. With just one window open, it will grab 60 to 90 megs of RAM (compared to, say, 14 for Chrome). With several pages open, which is normal for me, FireFox will grab 200 to 250 megs of RAM. Chrome might grab a 3rd of that. (These statements are true for FireFox 2 and 3, and Chrome 1.)

FireFox allows plugins, which is the main reason I use FireFox. The Firebug and Session Manager plugins are essential tools for me. However, FireFox doesn’t police the resource usage of these plugins. They can crash any machine: Macs, Linux, Windows. (Those of you who want to claim “Linux never crashes”, please note that a process can use up most of the memory on the machine, and then the machine becomes unresponsive. For the user, this is the same as a crash, even if in some hair-splitting way it avoids the technical definition of a crash.)

Chrome has the kind of plain, minimalist design that is a signature of most of Google’s products.  I like it a lot, though it has many annoyances. Yahoo Mail normally auto-suggests email addresses as I start to type them, but this doesn’t happen when I use Chrome. Also, when using WordPress, Chrome embeds inline styling, whereas other browsers do not. Also, again with WordPress, Chrome erases all line breaks every time I update a post, so that the text is reduced to one giant paragraph. Basically, most of the Javascript that is out there was not written with Chrome in mind, and Chrome has some kind of conflict with it. Also, surprisingly, I’m not able to log into some of my favorite forums with Chrome.  I get no error message, but I am not treated as logged in, even after giving the correct username and password (I have the same problem in Safari, but not in FireFox).

Scrolling a web page, using the arrow keys on the keyboard, is important to me. I read a great deal online, and for me it seems natural to want to use the arrow keys to move down the page as I read. Here is one area where Chrome is especially good. It scrolls smoothly. FireFox is usually broken in this regard – it tries to move the cursor down the page, but if the HTML is laid out in a way that allows the cursor to skip the main text of the page, then FireFox simply drops to the bottom of the page. This drives me crazy.

Safari 4 seems to be in between Chrome and FireFox in terms of resource use. It is wonderfully standards compliant and leads the way in supporting HTML5. I admire it for that, though until more browsers support HTML5, I can’t imagine using any of the new tags on a commercial web site.

Right now I can see using FireFox when I want to use my favorite plugins, and I can see using Chrome when I want a fast web browser, but I’m not sure what would cause me to use Safari.

I’m comfortable making this prediction: IE will continue to fade, and Microsoft will continue to fade, and FireFox, Safari and Chrome will all have more browser share a year from now than they currently do. So it is time for designer to start checking their designs in all of these browsers.

This is from Frank Rich’s column in the New York Times:

Much as Obama repealed the Bush restrictions on abortion and stem-cell research shortly after pushing through his stimulus package, so F.D.R. jump-started the repeal of Prohibition by asking Congress to legalize beer and wine just days after his March 1933 inauguration and declaration of a bank holiday. 

But F.D.R had to get a law passed through Congress, whereas President Obama is able to merely issue a decree. The comparison should remind us how much the power of the President has grown during the last 76 years.

These are the ads that Google thinks belong grouped together:

Seen over at Naked Capitalism.

An impressive effort: 

I was looking for a nice blogging solution for symfony, and all I found was a very simple plugin and a lot of people encouraging me to build my own blog. Even though it is a nice exercise, my philosophy is to not reinvent the wheel. Wordpress is surely the best free blogging tool available, so I preferred to spend time integrating it into my symfony application than to create yet another sfVeryEasyBlogPlugin.Integrating Wordpress into symfony can be done in three steps :

• integrating the blog into the application and its layout
• merging the authentification system
• integrating the blogging information into the symfony application

To be able to use WordPress inside a Symfony project, with an integrated login system, amounts to being able to extend WordPress with the infinite range allowed by Symfony – and that amounts to a major expansion of WordPress. The potential is great.

Lately I’ve noticed there have been less articles in the media about how the evil online social networks are corrupting the youth. danah boyd offer this bit of history:

While there were many adults on MySpace for legitimate purposes, it wasn’t until white collar professionals joined Facebook en masse that the moral panic started to subside. Finally, privileged Americans “got” social network sites, even if they were stuck confronting their high school identities through the listing of 25 things. At this stage, over 35% of American adults have a profile on a social network site. The adoption by this older, wealthier, more educated crowd changed the headlines of the news. Facebook became the new darling and most people thought that it had squashed MySpace long before it had even a fraction of the number of users.

Also, the features offered by a social site are not that important. With software like Photoshop and Word, more features might be a selling point (though that is frequently debated) but with social sites, more features might simply get in the way of what people really want to do. People go to these sites to socialize, and for that they don’t need a lot of features:

 Many who build technology think that a technology’s feature set is the key to its adoption and popularity. With social media, this is often not the case. There are triggers that drive early adopters to a site, but the single most important factor in determining whether or not a person will adopt one of these sites is whether or not it is the place where their friends hangout. In each of these cases, network effects played a significant role in the spread and adoption of the site.

Symfony relies on the Propel ORM (though in the newest version of Symfony, the Doctrine ORM is given equal status). Propel does not support every field type that MySql does. After some experimentation, I found that a MySql “datetime” field needs to be represented as a “timestamp” in Propel’s schema. 

Brad Sester catches this terrific bit of double-meaning:

From a Tuesday Wall Street Journal story on Citigroup’s efforts to raise common equity:

Citigroup officials hope to persuade private investors that have bought preferred shares — such as the Government of Singapore Investment Corp., Abu Dhabi Investment Authority and Kuwait Investment Authority — to follow the government’s lead in converting some of those stakes into common stock, according to people familiar with the matter.

Emphasis added

There is a difference between a minority stake held the investment arm of a government that doesn’t regulate a bank or backstop the banks’ liabilities and a large stake held by a banks’ home government. But it is striking that none of the “private investors” mentioned in the Wall Street Journal are actually, well, private investors. Sovereign wealth funds are at best a hybrid.

China for example has made it clear that it hopes that the US will protect at least some of its investments from the risk of losses. China’s Vice Premier Wang Qishan told Hank Paulson:

“We hope the US side will . . . guarantee the safety of China’s assets and investments in the US”

He may have just been thinking of the Agencies … but he also may have had a few of China’s other large stakes in mind. The classic response is that the only investment that is guaranteed by the full faith and credit of the United States government is a Treasury bond.

Yet, it increasingly looks like the US is inching toward severely diluting the common equity of a set of banks where sovereign funds have substantial stakes,* if not wiping out the existing equity entirely. That potentially — as Larry Summers warned in a former life — is a foreign policy issue.

Is it appropriate to refer to government agencies as “private investors”? What is the phrase “private investor” suppose to mean, if it includes entities such as  the Government of Singapore Investment Corp?

Last summer, when I was first learning Symfony, I had trouble figuring out how to do pagination in Symfony when I wrote a raw SQL query. I eventually resorted to creating database views, and then treating the views as if they were a normal database table, even to the point of creating model classes for them.

This would have been very handy, had I known it then:

Whatever you use, Propel or Doctrine, it is sometimes needed to write raw SQL queries.

$connection = Propel::getConnection();
$query = 'SELECT MAX(%s) AS max FROM %s';
$query = sprintf($query, ArticlePeer::CREATED_AT, ArticlePeer::TABLE_NAME);
$statement = $connection->prepare($query);
$statement->execute();
$resultset = $statement->fetch(PDO::FETCH_OBJ);
$max = $resultset->max;

When you do so, you loose all the relation object mapping. Besides, you cannot use any of the sf*Pager anymore.

Nevertheless, both Propel and Doctrine are based on PDO. It means that using raw queries doesn’t mean bypassing database abstraction. When you execute a query, you still retrieve a collection of PDOResulset objects, which are ‘abstract’ representations of database entries.

They’ve a useful looking new pager class listed over there.

For the sake of readability, I do this in my PHP code, as well:

One would definitely call it like this:

this.updatePost(true);

The problem is, that’s not very explanatory. In other words, you can’t tell straight from the line what this true does. I propose a solution that takes advantage of loose typing: just pass a string containing a parameter name as an argument.

this.updatePost('highlight');

Now, this is definitily more verbose.

I am dissatisfied with the HTML that is auto-generated by the built-in scaffolding system of Symfony.  I am planning to write a plugin that will generate HTML more to my liking. I know that, in the world of Ruby On Rails, there are several plugins that improve upon the built-in scaffolding system of that framework. Since these plugins have a longer history, and are more advanced, than anything in the world of Symfony, I am looking to them for some inspiration.

This is what Hobo says about itself:

It turns out that the hard part is not going fast, but staying flexible. This is where we think Hobo really shines. If you’ve played with “app builders” before, you’ll know about The Wall. The Wall is the point you reach where you have to give up and do it the old way because that one feature you really need just isn’t going to happen. Hobo doesn’t have one.

…Like Rails itself, Hobo encapsulates a number of opinions about web application development. The opinion that is most central to Hobo is this: we have found that there is a great deal of similarity from one web application to the next. Much more similarity in fact, than is supported by today’s frameworks. We would rather implement this stuff once.

Of course this approach is common to all frameworks—everything that Rails provides is there because many or all web applications will need it: database connectivity, session management, working with HTTP, etc. etc. The difference with Hobo is that we are trying to take this idea to a much higher level than is normally expected from a web framework. The ultimate goal is: don’t program your application, just declare it.

This is what ActiveScaffold says about itself:

Most web applications have many more model objects exposed on the backend, or admin side, than they do on the front. Coding interfaces for all those models is redundant and a waste of resources when all you need is CRUD functionality that’s smart enough to handle all your ActiveRecord associations.

Enter the ActiveScaffold plugin:

• An AJAXified table interface for creating, updating, and deleting objects
• Automatic handling of ActiveRecord associations
• Sorting, Search and Pagination
• Graceful JavaScript degradation
• RESTful API support (XML/YAML/JSON) baked in
• Sexy CSS styling and theming support
• More extension points than you can shake a stick at
• Guaranteed to work on Firefox 1+, IE 6+ and Safari 2+
• Released under the MIT License, the same one as Rails itself, so you can use it freely in your commercial applications.

This is what Streamlined says about itself:

Streamlined is a plugin for Ruby on Rails that provides an instant, production-ready UI for your ActiveRecord models. It started as a way to generate Administrative back-ends, but has become more general purpose and flexible over time. Streamlined aims to bring the declarative goodness of ActiveRecord to the view tier. It manages the presentation, creation, and editing of model instances, providing full-featured scaffolds equipped with everything you need to quickly and easily develop a rich user interface for interacting with your data, including:

• Zero-configuration relationship management
• Ajax-powered widgets and transitions
• Live data filtering
• Out-of-the-box sortable lists and pagination
• Exporting to XML, CSV, and JSON
• Declarative and easily-customizable UI development
• Cross-browser support including Firefox, Internet Explorer, and Safari
• Pluggable CSS styling

This gives a sense of the possibilities.

Hobo has an especially interesting goal, to create a framework that is declarative, rather than imperative.

In the short term, my goals for a new scaffolding system are simple:

1.) The HTML should use divs and CSS rather than tables.

2.) Pagination should be included by default.

3.) An administrative dashboard should be created by default.

I know I will want these 3 things from every Symfony project I start, so, for me at least, it makes sense to have  a plugin that automates all this.

I originally posted this over two years ago, on the Category4 blog. I want to keep a copy of this here, so I am reposting it. Nowadays I’m using Symfony for most of my projects, and Symfony does a lot of what I here criticize. I’ve come to the conclusion that:

1.) We must have a framework.

2.) It is too expensive to maintain our in-house framework, so we must adopt a well documented, open-source framework.

3.) All frameworks will have their stylistic quirks with which we will disagree.

4.) The benefits of using such frameworks will, in the long run, out weigh the negatives.

This is what I wrote two years ago (with one minor edit):

In my last post I tried to articulate some of the things I’d like to see in a PHP framework. Youngj posted a comment and suggested that I try Qcodo. I have a problem with Qcodo’s template system, and it is a problem that I have with a large number of other PHP frameworks: HTML that gets embedded in the PHP code (also known as PHP code that writes HTML). I’ll let this criticism of Qcodo stand-in for my criticism of all PHP frameworks that do this (which is most).

Please consider the example Qcodo gives on the front page of their site:

<? $this->RenderBegin(); ?>
Id: <? $this->txtId->Render(); ?>
Title: <? $this->txtTitle->Render(); ?>
<? $this->RenderEnd(); ?>

Bad. Not from a programming point of view, but from a work flow point of view, in a professional web design firm. In this example, the form is no longer being controlled by the designers. What happens if the designer decides that the Title should have a red background, but they don’t want to change the CSS values for all inputs on the page, so they need to give that particular input an id? They then need to come to me, the programmer, and say “Could you add an id to the Title input?” Thus, work is shifted from the designers to the programmers. This is bad economics, since in most web design shops the programmers are paid more than the designers. And why would I want extra work? More so, too many projects get delayed because the programming runs behind schedule. It is better to enable designers to do as much as possible on their own.

When the PHP code writes HTML, the programmers end up doing work which the designers can do better.

Much better would be something like this:

<form method=”post” action=”index.php”>
Id: <input type=”text” name=”id” value=”< ?= currentValue(”id”); ?>” />
Title: <input type=”text” name=”id” value=”< ?= currentValue(”title”); ?>” />
<input type=”submit” value=”Update this article?” />
</form>

This block of HTML is something the designer can control. What happens if the designer needs to add an id to any of these HTML elements? They can add it on their own, without bothering the programmer. And what happens if the client decides that the value of the submit button should be reworded? Who does the work go to? The designer or the programmer? Qcodo sends the work to the programmer, but in my example the work can go to the designer. That means Qcodo is bad economics, at least from the point of view of a professional web design firm.

I might ask why something like this is so popular among web application frameworks:

Title: <? $this->txtTitle->Render(); ?>

I believe that the programmers who write pages like this (and I used to be one of them) are hoping to achieve productivity gains by automating some of the work of creating forms. I believe this is a false economy. I don’t see that the number of characters is dramatically less in the Qcodo example than in my example. Whatever small gains Qcodo makes, they are lost as soon as a designer needs to come to a programmer to request a change.

I do believe these frameworks (such as Qcodo) achieve real efficiency gains if the programmer is working as a freelancer, and does both the programming and the designer work. If I was a single individual, unattached to any company, then I would probably use a framework with a template system just like Qcodo (in fact, I did, when I was just starting out). These template systems probably save programmers some time, but they waste time when non-programmers need to edit the pages. In a professional web design firm, one that handles large sites and therefore must have multiple people working on the site, I believe the best bet is to stick as close as possible to pure HTML. HTML is one of the few things that everyone in a web design firm will know.

Qcodo does a much better job of explaining itself and its philosophy than many other PHP frameworks. And from what I can tell, it seems to be well implemented. But it does not trust designers, and it writes too much HTML on it’s own. Consider the way it describes its code generation abilities:

Taking a quick look at the web front-end of codegen, you’ll see that only the top dozen or so lines (including comments) directly deal with the CodeGen object. Everything else past it is simply to paint a pretty HTML page to display the results.

“To paint a pretty HTML page”. I don’t think I’ll give it a try. I’ve a lot of PHP frameworks to test, and I’ve limited time, and this one sounds like it is taking too much power away from the designers.

If a programmer is also their own designer, then I can imagine why they’d want to write all the forms in PHP – it would save them some time. Peter Bowyer summed up that point of view with this post from 2003:

My current project is the biggest site I’ve had to design and build by myself. And in the process of planning the code I’ve realised why so many people have a framework they use. I’m having to do too much stuff here from scratch; code that should just plug in from existing projects. I’ve written a SQLBuilder library, polished up my own take on Fusebox, and am developing the FormProcessor library (the more I use it the more I realise why people do forms the “normal?? – i.e. writing PHP to construct them – way: it’s such a pain writing these forms by hand. Dreamweaver really doesn’t speed up the generation much.

Right. But if the forms do need to be edited by a designer, perhaps using Dreamweaver, then it is better to have the form elements defined as pure HTML. If you are a lone, individual programmer you can do everything to suit your own taste, but when you’re working in a shop where several people work together on one site, then one needs a framework that maximizes how much the designers can do on their own.

I am reading Warren Buffet’s annual letter to his the shareholders of Berkshire Hathaway. I am surprised to like the writing so much. I like the candor, and I also like the humor, which appears in clever turns of phrase, in almost every paragraph. I did not expect to find this letter interesting, but in fact, I was sucked in and read the whole thing. It is one of the most interesting essays about the American economy that I’ve read in awhile. A few excerpts that intrigued me:

By yearend, investors of all stripes were bloodied and confused, much as if they were small birds that had strayed into a badminton game.

As the year progressed, a series of life-threatening problems within many of the world’s great financial institutions was unveiled. This led to a dysfunctional credit market that in important respects soon turned non-functional. The watchword throughout the country became the creed I saw on restaurant walls when I was young: “In God we trust; all others pay cash.”

……

Economic medicine that was previously meted out by the cupful has recently been dispensed by the barrel. These once-unthinkable dosages will almost certainly bring on unwelcome aftereffects. Their precise nature is anyone’s guess, though one likely consequence is an onslaught of inflation. Moreover, major industries have become dependent on Federal assistance, and they will be followed by cities and states bearing mind-boggling requests.

……

There’s another less pleasant reality: During 2008 I did some dumb things in investments. I made at least one major mistake of commission and several lesser ones that also hurt. I will tell you more about these later. Furthermore, I made some errors of omission, sucking my thumb when new facts came in that should have caused me to re-examine my thinking and promptly take action.

……

Our long-avowed goal is to be the “buyer of choice” for businesses – particularly those built and owned by families. The way to achieve this goal is to deserve it. That means we must keep our promises; avoid leveraging up acquired businesses; grant unusual autonomy to our managers; and hold the purchased companies through thick and thin (though we prefer thick and thicker).

Our record matches our rhetoric. Most buyers competing against us, however, follow a different path. For them, acquisitions are “merchandise.” Before the ink dries on their purchase contracts, these operators are contemplating “exit strategies.” We have a decided advantage, therefore, when we encounter sellers who truly care about the future of their businesses.

Some years back our competitors were known as “leveraged-buyout operators.” But LBO became a bad name. So in Orwellian fashion, the buyout firms decided to change their moniker. What they did not change,though, were the essential ingredients of their previous operations, including their cherished fee structures and love of leverage. Their new label became “private equity,” a name that turns the facts upside-down: A purchase of a business by these firms almost invariably results in dramatic reductions in the equity portion of the acquiree’s capital structure compared to that previously existing. A number of these acquirees, purchased only two to three years ago, are now in mortal danger because of the debt piled on them by their private-equity buyers.

Much of the bank debt is selling below 70¢ on the dollar, and the public debt has taken a far greater beating. The private equity firms, it should be noted, are not rushing in to inject the equity their wards now desperately need. Instead,they’re keeping their remaining funds very private.

…….

Our insurance group has propelled Berkshire’s growth since we first entered the business in 1967. This happy result has not been due to general prosperity in the industry. During the 25 years ending in 2007, return on net worth for insurers averaged 8.5% versus 14.0% for the Fortune 500. Clearly our insurance CEOs have not had the wind at their back. Yet these managers have excelled to a degree Charlie and I never dreamed possible in the early days. Why do I love them? Let me count the ways.

At GEICO, Tony Nicely – now in his 48th year at the company after joining it when he was 18 –continues to gobble up market share while maintaining disciplined underwriting. When Tony became CEO in 1993, GEICO had 2.0% of the auto insurance market, a level at which the company had long been stuck. Now we have a 7.7% share, up from 7.2% in 2007.

The combination of new business gains and an improvement in the renewal rate on existing businesshas moved GEICO into the number three position among auto insurers. In 1995, when Berkshire purchased control, GEICO was number seven. Now we trail only State Farm and Allstate. GEICO grows because it saves money for motorists. No one likes to buy auto insurance. But virtuallyeveryone likes to drive. So, sensibly, drivers look for the lowest-cost insurance consistent with first-class service.Efficiency is the key to low cost, and efficiency is Tony’s specialty. Five years ago the number of policies peremployee was 299. In 2008, the number was 439, a huge increase in productivity.

…As we view GEICO’s current opportunities, Tony and I feel like two hungry mosquitoes in a nudist camp. Juicy targets are everywhere. First, and most important, our new business in auto insurance is now exploding. Americans are focused on saving money as never before, and they are flocking to GEICO. In January 2009, we set a monthly record – by a wide margin – for growth in policy holders. That record will last exactly 28 days: As we go to press, it’s clear February’s gain will be even better.

……

I will write here at some length about the mortgage operation of Clayton Homes and skip any financial commentary, which is summarized in the table at the end of this section. I do this because Clayton’s recent experience may be useful in the public-policy debate about housing and mortgages. But first a little background. Clayton is the largest company in the manufactured home industry, delivering 27,499 units last year.

This came to about 34% of the industry’s 81,889 total. Our share will likely grow in 2009, partly because much of the rest of the industry is in acute distress. Industrywide, units sold have steadily declined since they hit a peak of 372,843 in 1998.

…This 1997-2000 fiasco should have served as a canary-in-the-coal-mine warning for the far-larger conventional housing market. But investors, government and rating agencies learned exactly nothing from the manufactured-home debacle. Instead, in an eerie rerun of that disaster, the same mistakes were repeated with conventional homes in the 2004-07 period: Lenders happily made loans that borrowers couldn’t repay out of their incomes, and borrowers just as happily signed up to meet those payments. Both parties counted on “house-price appreciation” to make this otherwise impossible arrangement work. It was Scarlett O’Hara all over again: “I’ll think about it tomorrow.” The consequences of this behavior are now reverberating through every corner of our economy.

Clayton’s 198,888 borrowers, however, have continued to pay normally throughout the housing crash, handing us no unexpected losses. This is not because these borrowers are unusually creditworthy, a point proved by FICO scores (a standard measure of credit risk). Their median FICO score is 644, compared to a national median of 723, and about 35% are below 620, the segment usually designated “sub-prime.” Many disastrous pools of mortgages on conventional homes are populated by borrowers with far better credit, as measured by FICO scores.

Yet at yearend, our delinquency rate on loans we have originated was 3.6%, up only modestly from 2.9% in 2006 and 2.9% in 2004. (In addition to our originated loans, we’ve also bought bulk portfolios of various types from other financial institutions.) Clayton’s foreclosures during 2008 were 3.0% of originated loans compared to 3.8% in 2006 and 5.3% in 2004.

Why are our borrowers – characteristically people with modest incomes and far-from-great credit scores – performing so well? The answer is elementary, going right back to Lending 101. Our borrowers simply looked at how full-bore mortgage payments would compare with their actual – not hoped-for – income and then decided whether they could live with that commitment. Simply put, they took out a mortgage with the intention of paying it off, whatever the course of home prices.

Just as important is what our borrowers did not do. They did not count on making their loan payments by means of refinancing. They did not sign up for “teaser” rates that upon reset were outsized relative to their income. And they did not assume that they could always sell their home at a profit if their mortgage payments became onerous. Jimmy Stewart would have loved these folks.

…….

[On insuring tax exempt bonds]

The rationale behind very low premium rates for insuring tax-exempts has been that defaults have historically been few. But that record largely reflects the experience of entities that issued uninsured bonds. Insurance of tax-exempt bonds didn’t exist before 1971, and even after that most bonds remained uninsured.

A universe of tax-exempts fully covered by insurance would be certain to have a somewhat different loss experience from a group of uninsured, but otherwise similar bonds, the only question being how different. To understand why, let’s go back to 1975 when New York City was on the edge of bankruptcy. At the time its bonds – virtually all uninsured – were heavily held by the city’s wealthier residents as well as by New York banks and other institutions. These local bondholders deeply desired to solve the city’s fiscal problems. So before long, concessions and cooperation from a host of involved constituencies produced a solution. Without one, it was apparent to all that New York’s citizens and businesses would have experienced widespread and severe financial losses from their bond holdings.

Now, imagine that all of the city’s bonds had instead been insured by Berkshire. Would similar belttightening, tax increases, labor concessions, etc. have been forthcoming? Of course not. At a minimum, Berkshire would have been asked to “share” in the required sacrifices. And, considering our deep pockets, the required contribution would most certainly have been substantial.

Local governments are going to face far tougher fiscal problems in the future than they have to date. The pension liabilities I talked about in last year’s report will be a huge contributor to these woes. Many cities and states were surely horrified when they inspected the status of their funding at yearend 2008. The gap between assets and a realistic actuarial valuation of present liabilities is simply staggering.

When faced with large revenue shortfalls, communities that have all of their bonds insured will be more prone to develop “solutions” less favorable to bondholders than those communities that have uninsured bonds held by local banks and residents. Losses in the tax-exempt arena, when they come, are also likely to be highly correlated among issuers. If a few communities stiff their creditors and get away with it, the chance that others will follow in their footsteps will grow. What mayor or city council is going to choose pain to local citizens in the form of major tax increases over pain to a far-away bond insurer?

…..

The type of fallacy involved in projecting loss experience from a universe of non-insured bonds onto a deceptively-similar universe in which many bonds are insured pops up in other areas of finance. “Back-tested” models of many kinds are susceptible to this sort of error. Nevertheless, they are frequently touted in financial markets as guides to future action. (If merely looking up past financial data would tell you what the future holds, the Forbes 400 would consist of librarians.)

Indeed, the stupefying losses in mortgage-related securities came in large part because of flawed, history-based models used by salesmen, rating agencies and investors. These parties looked at loss experience over periods when home prices rose only moderately and speculation in houses was negligible. They then made this experience a yardstick for evaluating future losses. They blissfully ignored the fact that house prices had recently skyrocketed, loan practices had deteriorated and many buyers had opted for houses they couldn’t afford. In short, universe “past” and universe “current” had very different characteristics. But lenders, government and media largely failed to recognize this all-important fact.

……

I told you in an earlier part of this report that last year I made a major mistake of commission (and maybe more; this one sticks out). Without urging from Charlie or anyone else, I bought a large amount of ConocoPhillips stock when oil and gas prices were near their peak. I in no way anticipated the dramatic fall in energy prices that occurred in the last half of the year. I still believe the odds are good that oil sells far higher in the future than the current $40-$50 price. But so far I have been dead wrong. Even if prices should rise, moreover, the terrible timing of my purchase has cost Berkshire several billion dollars.

I made some other already-recognizable errors as well. They were smaller, but unfortunately not that small. During 2008, I spent $244 million for shares of two Irish banks that appeared cheap to me. At yearend we wrote these holdings down to market: $27 million, for an 89% loss. Since then, the two stocks have declined even further. The tennis crowd would call my mistakes “unforced errors.”

……

The investment world has gone from underpricing risk to overpricing it. This change has not been minor; the pendulum has covered an extraordinary arc. A few years ago, it would have seemed unthinkable that yields like today’s could have been obtained on good-grade municipal or corporate bonds even while risk-free governments offered near-zero returns on short-term bonds and no better than a pittance on long-terms. When the financial history of this decade is written, it will surely speak of the Internet bubble of the late 1990s and the housing bubble of the early 2000s. But the U.S. Treasury bond bubble of late 2008 may be regarded as almost equally extraordinary.

Clinging to cash equivalents or long-term government bonds at present yields is almost certainly a terrible policy if continued for long. Holders of these instruments, of course, have felt increasingly comfortable – in fact, almost smug – in following this policy as financial turmoil has mounted. They regard their judgment confirmed when they hear commentators proclaim “cash is king,” even though that wonderful cash is earning close to nothing and will surely find its purchasing power eroded over time.

Approval, though, is not the goal of investing. In fact, approval is often counter-productive because it sedates the brain and makes it less receptive to new facts or a re-examination of conclusions formed earlier. Beware the investment activity that produces applause; the great moves are usually greeted by yawns.

…….

Derivatives are dangerous. They have dramatically increased the leverage and risks in our financial system. They have made it almost impossible for investors to understand and analyze our largest commercial banks and investment banks. They allowed Fannie Mae and Freddie Mac to engage in massive misstatements of earnings for years. So indecipherable were Freddie and Fannie that their federal regulator, OFHEO, whose more than 100 employees had no job except the oversight of these two institutions, totally missed their cooking of the books.

Indeed, recent events demonstrate that certain big-name CEOs (or former CEOs) at major financial institutions were simply incapable of managing a business with a huge, complex book of derivatives. Include Charlie and me in this hapless group: When Berkshire purchased General Re in 1998, we knew we could not get our minds around its book of 23,218 derivatives contracts, made with 884 counterparties (many of which we had never heard of). So we decided to close up shop. Though we were under no pressure and were operating in benign markets as we exited, it took us five years and more than $400 million in losses to largely complete the task. Upon leaving, our feelings about the business mirrored a line in a country song: “I liked you better before I got to know you so well.”

Improved “transparency” – a favorite remedy of politicians, commentators and financial regulators for averting future train wrecks – won’t cure the problems that derivatives pose. I know of no reporting mechanism that would come close to describing and measuring the risks in a huge and complex portfolio of derivatives. Auditors can’t audit these contracts, and regulators can’t regulate them. When I read the pages of “disclosure” in 10-Ks of companies that are entangled with these instruments, all I end up knowing is that I don’t know what is going on in their portfolios (and then I reach for some aspirin).

A normal stock or bond trade is completed in a few days with one party getting its cash, the other its securities. Counterparty risk therefore quickly disappears, which means credit problems can’t accumulate. This rapid settlement process is key to maintaining the integrity of markets. That, in fact, is a reason for NYSE and NASDAQ shortening the settlement period from five days to three days in 1995.

Derivatives contracts, in contrast, often go unsettled for years, or even decades, with counterparties building up huge claims against each other. “Paper” assets and liabilities – often hard to quantify – become important parts of financial statements though these items will not be validated for many years. Additionally, a frightening web of mutual dependence develops among huge financial institutions. Receivables and payables by the billions become concentrated in the hands of a few large dealers who are apt to be highly-leveraged in other ways as well. Participants seeking to dodge troubles face the same problem as someone seeking to avoid venereal disease: It’s not just whom you sleep with, but also whom they are sleeping with.

……

We endorse mark-to-market accounting. I will explain later, however, why I believe the Black-Scholes formula, even though it is the standard for establishing the dollar liability for options, produces strange results when the long-term variety are being valued.

…The ridiculous premium that Black-Scholes dictates in my extreme example is caused by the inclusion of volatility in the formula and by the fact that volatility is determined by how much stocks have moved around in some past period of days, months or years. This metric is simply irrelevant in estimating the probability weighted range of values of American business 100 years from now. (Imagine, if you will, getting a quote every day on a farm from a manic-depressive neighbor and then using the volatility calculated from these changing quotes as an important ingredient in an equation that predicts a probability-weighted range of values for the farm a century from now.)