unfortunate reality to CMS based web development

An unfortunate reality to CMS based web development is that in most cases developers are used to working alone.  When working with a CMS most of the programming involved is customization rather than from scratch development, so the need to involve multiple programmers as well as manage and coordinate a development team is minimized.

When a team is involved, a completely different work flow is involved.  Proper communication channels, development procedures, code testing constructs, naming schemes, and rules for system architecture must be in place in order for a programming team to work together effectively.  Moreover, developers must be well versed in these rules to work in that type of environment effectively.

The rules of a CMS are built to work within that particular system.  Isolating modules into packaged pieces that can be installed with the click of a button is a common mode of operation.  Within a sound application architecture, the goal is to minimize code replication so that entire logical systems can be altered from a single location over time to prevent maintenance time from growing at the same rate that the site grows.


Database industry is growing fifty percent faster than the software market



Revenue from software for structured data management and data access, analysis and delivery grew 7.3 percent and 6 percent, respectively in 2013, according to IDC. Both sectors grew faster than the software market as a whole, which grew 5.5 percent year over year.


DevCoin – cryptocurrency to support open source development

Devcoin is an open source project designed to support open source projects through the use of a dedicated crypto currency.

Crypto currency is a new form of digital money which is decentralized and runs on a distributed network of computers. Cryptographic techniques are used to secure the network and each individual’s funds, while open source code provides the shared protocol through which each wallet or service communicates with the public “block chain” where information on all transactions is stored. The most famous example of this is, of course, Bitcoin.


Unlike traditional fiat currencies, which are effectively created by banks and issued into the economy through interest-bearing debt instruments, newly created coins from cryptocurrency are generally distributed to their community of users. There are many ways to do this. Bitcoin uses “Proof of Work” which rewards “miners” who run software which is essential for maintaining the network. Other coins use proof of stake to reward people who have purchased coins or other exotic methods.

An ethical cryptocurrency

Devcoin takes a different approach. Only 10% of newly created Devcoins (DVC) are paid out to miners, through merged mining alongside Bitcoin (meaning that people can mine both Bitcoin and Devcoin on the same machine, at the same time). This is sufficient to secure the network, but leaves 90% of newly minted coins left over, which can then be given out for free to open source developers—and also creative commons media producers.

First created in 2011 as a fork of Bitcoin, Devcoin was the first “ethical cryptocurrency” whose sole purpose is to support open source development and creative commons media. It is now one of the longest-running block chain based digital currencies.

Payouts and bounties

Newly created coins are paid out in the form of bounties, designed to encourage the development of a wide range of open source projects from new software, to hardware products such as new forms of 3D printer. There are even tentative future plans to finance a bounty for the development of a single seater open source spacecraft capable of exiting and re-entering the earth’s atmosphere! Developers can also earn DVC through the Devcoin and Bitcoin share lists, which reward active open source developers within the cryptocurrency community.

New web services are also under development which will allow anybody to create or to help fund bounties for new projects.

Writers can also earn Devcoins by contributing articles to the Devtome wiki. All work submitted to Devtome is published under a Creative Commons license. Quality is maintained through a ratings system, and writers’ compensation is calculated according to their quality rating and the number of words published.

Earned Devcoins can then be spent at a small but growing range of online retailers, or changed for other currencies through an exchange.

Donating to worthy open source projects

Because many people buy Devcoins or get involved a project which will lead to them earning these coins because they believe in the power of open source development and want to support it, there is also a community of DVC holders willing to donate their coins to worthy open source projects. The simplest way to use the coin to support a new or established project is therefore to accept DVC donations and to let the community know through the official forum.

For people looking to use Devcoin to help support projects that they believe in, rather than to finance a project, all that is required is to buy (or earn) some Devcoins and then spend or donate them! Because of the “network effect” of digital currencies, a larger number of users means a rising value per coin, as demand is raised relative to the fixed supply—which in turn increases the value of the new coins paid out to support open source developers.

Devcoin is a fully open source cryptocurrency. For more information, check out the official Devcoin site at devcoin.org.

Microsoft Tackles 'SQL Server Sprawl'

Hewlett-Packard has partnered with Microsoft to introduce a preconfigured system for companies that want to perform analytics on data spread across many SQL Server databases.

The HP ConvergedSystem 300 for the Microsoft Analytics Platform can draw structured data from SQL Servers and unstructured data from other sources.

Diverse data sets can be brought into Microsoft’s SQL Server Parallel Data Warehouse or the data can be queried from its source.

All of this is handled through Hadoop, an open source framework that lets enterprises store and process Big Data across commodity server clusters.

Once the data is ready, it can be analyzed with applications capable of finding actionable business intelligence (BI). Software tailor made for the configuration includes Microsoft Power BI for Office 365 enterprise subscribers.

The HP product, introduced May 8, is designed for midsize and large companies dealing with “SQL sprawl,” Frances Guida, manager of HP ConvergedSystems, told CruxialCIO.

“They’ve got data in multiple databases deployed across different businesses and it’s very hard to really get that kind of (single) view,” Guida said.

HP ConvergedSystems comprise integrated software and hardware, including storage, networking and servers. It’s all optimized for specific workloads, such as virtualization, cloud computing and, in the case of the latest product, Big Data.

The latter is a term used for a collection of data sets so large and complex that they couldn’t be processed using standard database management tools or traditional data processing applications.

Revenue from software for structured data management and data access, analysis and delivery grew 7.3 percent and 6 percent, respectively in 2013, according to IDC. Both sectors grew faster than the software market as a whole, which grew 5.5 percent year over year.

The Drawback to Web Frameworks

Web frameworks are great, don’t get me wrong here. They provide a structure and consistency across projects that will transcend developers over the life of a system while dramatically simplifying the code base amongst other wonderful side effects. But what’s the downside?

(imagine my best old man voice)

Pull up a chair sonny and let me get my old man cane out of the way so I can tell you what web development looked like back in my day.

Back when I was coming up you had to write the frameworks yourself! It was a rite of passage for every PHP developer out there and everybody thought their framework was better than everybody else’s.  That hasn’t really stopped either as PHP is saturated with frameworks these days. Most of them are modeled in one form or another, after Ruby on Rail’s MVC approach too.

Before all of these different frameworks existed web developers tended to be SQL junkies writing crazy, page long queries that would put the pieces of their heavily normalized databases back together.  Data management logic would actually live in the database itself in the form of SQL functions, triggers and stored procedures.

Then frameworks came along.  You didn’t have to “mess with” all of that ugly SQL anymore.

There in lies the problem and the project I just finished up is a perfect example of it.  I’ve never seen a project handled badly by somebody who understood databases.  The database(s) is the backbone of every major web application in existence.

The problem with the advent of the framework era is that rather than taking advantage of all of those extremely powerful tools under the hood of their database, many developers avoid them like the plague. Simply because they haven’t ever been forced to work directly with the database thanks to the framework hiding so much of it from them.

That’s the issue in totality. Databases exist to manage data, not simply to be a dumping ground for you to save and retrieve things from. That’s what a file system does.

Databases have tools built in to handle statistical analysis, complex mathematics, caching, triggered actions based on data changes, transactions, flexible full text searches, specialized data types that allow for processing and traversing XML, data compression and that’s barely scratching the surface.

But if you were to ask most developers who have learned with frameworks, you’d be lucky if they could tell you the difference between a LEFT JOIN and an INNER JOIN.  It’s maddening.

I just finished spending a year cleaning up a mess left by some very good Ruby on Rails developers. The problem is that Ruby on Rails was where their talent began an ended. The mess left by these guys because they didn’t have any idea what their database could do was so bad that it almost sank a company. There were pages that you could visit which would crash the site. In order to look up what the status was on a particular object related to a user, they used some beautiful looking Rails code to find, filter, and combine results (and THEN paginate). The problem is that after Rails goes one level deep from a single record it starts performing single queries for each record in each relationship. That meant, for example, that in order to retrieve the most recent objects for a user who had over 18,000 in his account history that upwards of 50,000 queries were executed.  The results of those 50,000 queries were then loaded into the web server’s RAM (and SWAP), processed/sorted/filtered, and THEN paginated just to show the first 100 results.  It was appalling and that is only one example.

Data management logic belongs in the database in most cases.  There are exceptions of course but understanding how the data needs to be processed and the constraints involved will allow you to identify what those exceptions are going to be as well as how to handle them.

Putting data management logic in your application code base is probably the #1 cause of race conditions as well. That unique check on your model’s before save hook is not always going to work. The unique index on your database will.

Please, for the love of all that is good about the web; if you are a web developer who’s come up on frameworks and don’t feel like you could write complex SQL with your eyes closed…LEARN! Please! If you have questions, post them in a comment on this page and I’ll be happy to take a look.

That database backing your system is a powerful thing.  Don’t just ignore it, take advantage of it.  That’s the biggest drawback to web frameworks: they teach developers to ignore it.


The Sonics are coming home??

It looks like Steve Ballmer is starting to purchase the LA Clippers. I would be SOOOOO excited to have the Seattle SuperSonics back in town.  How often does a team in the playoffs get sold??

More importantly.. What does BASKETBALL have to do with databases and Visual Basic? (Stay tuned)

I know that Ballmer.. And Microsoft has a bad reputation. I don’t care.

I always loved Ballmer.  I just wish that Bill Gates never left.. Because then Visual Basic wouldn’t have gotten the short end of the stick.  It still breaks my heart that VB was the most popular language in the world just eleven years ago.

I don’t think that VB died because of OOP.  It died because of a marketing problem created by Sun.. And it is a tragedy that to this day there are about ten different roles that Visual Basic can fulfill..  C#  seems like such a half-baked solution to me.

Is C# ever going to support multiple inheritance? I honestly insist that Visual Basic will make a comeback.. JUST LIKE OUR SONICS… I can’t wait until Visual Basic comes home.


mySQL Scalability on Amadon RDS

Today, I’d like to discuss getting better MySQL scalability on Amazon RDS.

The question of the day: “What can you do when a MySQL database needs to scale write-intensive workloads beyond the capabilities of the largest available machine on Amazon RDS?”

Let’s take a look.

In a typical EC2/RDS set-up, users connect to app servers from their mobile devices and tablets, computers, browsers, etc.  Then app servers connect to an RDS instance (web/cloud services) and in some cases they might leverage some read-only replicas.


Figure 1. A typical RDS instance is a single-instance database, with read replicas.  This is not very good at handling high write-based throughput.

As your application becomes more popular you can expect an increasing number of users, more transactions, and more accumulated data.  User interactions can become more challenging as the application adds more sophisticated capabilities. The result of all this positive activity: your MySQL database will inevitably begin to experience scalability pressures.

What can you do?

Broadly speaking, there are four options available to improve MySQL scalability on RDS.

1. Larger RDS Instances – If you’re not already using the maximum available RDS instance, you can always scale up – to larger hardware.  Bigger CPUs, more compute power, more memory et cetera. But the largest available RDS instance is still limited.  And they get expensive.

“High-Memory Quadruple Extra Large DB Instance”:

  • 68 GB of memory
  • 26 ECUs (8 virtual cores with 3.25 ECUs each)
  • 64-bit platform
  • High I/O Capacity
  • Provisioned IOPS Optimized: 1000Mbps

2. Provisioned IOPs – You can get provisioned IOPs and higher throughput on the I/O level.

However, there is a hard limit with a maximum instance size and maximum number ofprovisioned IOPs you can buy from Amazon and you simply cannot scale beyond these hardware specifications.

3. Leverage Read Replicas – If your application permits, you can leverage read replicas to offload some reads from the master databases. But there are a limited number of replicas you can utilize and Amazon generally requires some modifications to your existing application.

And read-replicas don’t help with write-intensive applications.

4. Multiple Database Instances – Amazon offers a fourth option:

You can implement partitioning,thereby spreading your data across multiple database Instances” (Link)

However, Amazon does not offer any guidance or facilities to help you with this. “Multiple database instances” is not an RDS feature.  And Amazon doesn’t explain how to implement this idea.

In fact, when asked, this is the response on an Amazon forum:

Q: Is there any documents that describe the partition DB across multiple RDS?
I need to use DB with more 1TB but exist a limitation during the create process, but I read in the any FAQ that you need to partition database, but I don’t find any documents that describe it.

A: “DB partitioning/sharding is not an official feature of Amazon RDS or MySQL, but a technique to scale out database by using multiple database instances. The appropriate way to split data depends on the characteristics of the application or data set. Therefore, there is no concrete and specific guidance.”

So now what?

The answer is to scale out with ScaleBase.

Amazon RDS with ScaleBase: What you get – MySQL Scalability!

ScaleBase is specifically designed to scale out a single MySQL RDS instance into multiple MySQL instances.

Critically, this is accomplished with no changes to your application code.  Your application continues to “see” one database.   ScaleBase does all the work of managing and enforcing an optimized data distribution policy to create multiple MySQL instances.

With ScaleBase, data distribution, transactions, concurrency control, and two-phase commit are all 100% transparent and 100% ACID-compliant, so applications, services and tooling continue to interact with your distributed RDS as if it were a single MySQL instance.

The result: now you can cost-effectively leverage multiple MySQL RDS instance to scale out write-intensive workloads to an unlimited number of users, transactions, and data.

Amazon RDS with ScaleBase: What you keep – Everything!

And how does this change your Amazon environment?

1. Keep your application, unchanged – There is no change your application development life-cycle at all.  You still use your existing development tools, frameworks and libraries.  Application quality assurance and testing cycles stay the same. And, critically, you stay withan ACID-compliant MySQL environment.

2. Keep your RDS value-added services – The value-added services that you rely on are all still available. Amazon will continue to handle database maintenance and updates for you. You can still leverage High Availability via Multi A-Z.  And, if it benefits youra application throughput, you can still use read replicas.

3. Keep your RDS administration – Finally the RDS monitoring and provisioning tools you rely on still work as they did before.

With your one large MySQL instance, now split into multiple instances, you can actually use less expensive, smallersmaller available RDS hardware and continue to see better database performance.


Amazon RDS is a tremendous service, but it doesn’t offer solutions to scale beyond a single MySQL instance. Larger RDS instances get more expensive.  And when you max-out on the available hardware, you’re stuck.  Amazon recommends scaling out your single instance into multiple instances for transaction-intensive apps, but offers no services or guidance to help you. This is where ScaleBase comes in to save the day.

It gives you a simple and effective way to create multiple MySQL RDS instances, while removing all the complexities typically caused by “DIY” sharding andwith no changes to your applications .

With ScaleBase you continue to leverage the AWS/RDS ecosystem: commodity hardware and value added services like read replicas, multi A-Z, maintenance/updates and administration with monitoring tools and provisioning.


If you’re curious to try ScaleBase on Amazon, it can be found here – Download NOW.

SQL Server creates a histogram on the leading key but not on the other columns

the leading key should be the most selective column—number in our case. Allegedly (according to the recommendation), SQL Server creates a histogram on the leading key but not on the other columns; so if you follow the recommendation, the optimizer can produce better cardinality estimates, which tend to lead to more optimal choices.


Supratimas – analysis tool for Microsoft SQL Server query execution plans

I found the GREATEST tool ever.. for analyzing SQL Server Query Plans.  I can’t wait to spend some more time with this!


Color Coding
Nodes with cost of more than one per cent are colored blue. That makes it easier to find them.

Display number of rows and total size of data produced by the node.

Easily move between nodes using navigation buttons. Center on node by double-click.

Incoming Cost
Show the cost of incoming subtrees. You can optionally collapse a subtree that is of no interest to you.

In-place properties
Show the cost of incoming subtrees. You can optionally collapse subtree that has no interest to you.

Rule-based warnings
Supratimas has a customizable, rule-based warning system. It can point to potential problems that lead to performance problems.

Warnings Summary
List of all warnings with an ability to navigate to the node.

Expensive Operations Summary
List of all nodes that have a cost of more than one cent.